{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 个性化推荐\n",
    "本项目使用卷积神经网络，并使用[`MovieLens`](https://grouplens.org/datasets/movielens/)数据集完成电影推荐的任务。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "推荐系统在日常的网络应用中无处不在，比如网上购物、网上买书、新闻app、社交网络、音乐网站、电影网站等等等等，有人的地方就有推荐。根据个人的喜好，相同喜好人群的习惯等信息进行个性化的内容推荐。比如打开新闻类的app，因为有了个性化的内容，每个人看到的新闻首页都是不一样的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "这当然是很有用的，在信息爆炸的今天，获取信息的途径和方式多种多样，人们花费时间最多的不再是去哪获取信息，而是要在众多的信息中寻找自己感兴趣的，这就是信息超载问题。为了解决这个问题，推荐系统应运而生。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "协同过滤是推荐系统应用较广泛的技术，该方法搜集用户的历史记录、个人喜好等信息，计算与其他用户的相似度，利用相似用户的评价来预测目标用户对特定项目的喜好程度。优点是会给用户推荐未浏览过的项目，缺点呢，对于新用户来说，没有任何与商品的交互记录和个人喜好等信息，存在冷启动问题，导致模型无法找到相似的用户或商品。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "为了解决冷启动的问题，通常的做法是对于刚注册的用户，要求用户先选择自己感兴趣的话题、群组、商品、性格、喜欢的音乐类型等信息，比如豆瓣FM：\n",
    "<img src=\"assets/IMG_6242_300.PNG\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 下载数据集\n",
    "运行下面代码把[`数据集`](http://files.grouplens.org/datasets/movielens/ml-1m.zip)下载下来"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "f:\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "import numpy as np\n",
    "from collections import Counter\n",
    "import tensorflow as tf\n",
    "\n",
    "import os\n",
    "import pickle\n",
    "import re\n",
    "from tensorflow.python.ops import math_ops\n",
    "\n",
    "from urllib.request import urlretrieve\n",
    "from os.path import isfile, isdir\n",
    "from tqdm import tqdm\n",
    "import zipfile\n",
    "import hashlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "def _unzip(save_path, _, database_name, data_path):\n",
    "    \"\"\"\n",
    "    Unzip wrapper with the same interface as _ungzip\n",
    "    :param save_path: The path of the gzip files\n",
    "    :param database_name: Name of database\n",
    "    :param data_path: Path to extract to\n",
    "    :param _: HACK - Used to have to same interface as _ungzip\n",
    "    \"\"\"\n",
    "    print('Extracting {}...'.format(database_name))\n",
    "    with zipfile.ZipFile(save_path) as zf:\n",
    "        zf.extractall(data_path)\n",
    "\n",
    "def download_extract(database_name, data_path):\n",
    "    \"\"\"\n",
    "    Download and extract database\n",
    "    :param database_name: Database name\n",
    "    \"\"\"\n",
    "    DATASET_ML1M = 'ml-1m'\n",
    "\n",
    "    if database_name == DATASET_ML1M:\n",
    "        url = 'http://files.grouplens.org/datasets/movielens/ml-1m.zip'\n",
    "        hash_code = 'c4d9eecfca2ab87c1945afe126590906'\n",
    "        extract_path = os.path.join(data_path, 'ml-1m')\n",
    "        save_path = os.path.join(data_path, 'ml-1m.zip')\n",
    "        extract_fn = _unzip\n",
    "\n",
    "    if os.path.exists(extract_path):\n",
    "        print('Found {} Data'.format(database_name))\n",
    "        return\n",
    "\n",
    "    if not os.path.exists(data_path):\n",
    "        os.makedirs(data_path)\n",
    "\n",
    "    if not os.path.exists(save_path):\n",
    "        with DLProgress(unit='B', unit_scale=True, miniters=1, desc='Downloading {}'.format(database_name)) as pbar:\n",
    "            urlretrieve(\n",
    "                url,\n",
    "                save_path,\n",
    "                pbar.hook)\n",
    "\n",
    "    assert hashlib.md5(open(save_path, 'rb').read()).hexdigest() == hash_code, \\\n",
    "        '{} file is corrupted.  Remove the file and try again.'.format(save_path)\n",
    "\n",
    "    os.makedirs(extract_path)\n",
    "    try:\n",
    "        extract_fn(save_path, extract_path, database_name, data_path)\n",
    "    except Exception as err:\n",
    "        shutil.rmtree(extract_path)  # Remove extraction folder if there is an error\n",
    "        raise err\n",
    "\n",
    "    print('Done.')\n",
    "    # Remove compressed data\n",
    "#     os.remove(save_path)\n",
    "\n",
    "class DLProgress(tqdm):\n",
    "    \"\"\"\n",
    "    Handle Progress Bar while Downloading\n",
    "    \"\"\"\n",
    "    last_block = 0\n",
    "\n",
    "    def hook(self, block_num=1, block_size=1, total_size=None):\n",
    "        \"\"\"\n",
    "        A hook function that will be called once on establishment of the network connection and\n",
    "        once after each block read thereafter.\n",
    "        :param block_num: A count of blocks transferred so far\n",
    "        :param block_size: Block size in bytes\n",
    "        :param total_size: The total size of the file. This may be -1 on older FTP servers which do not return\n",
    "                            a file size in response to a retrieval request.\n",
    "        \"\"\"\n",
    "        self.total = total_size\n",
    "        self.update((block_num - self.last_block) * block_size)\n",
    "        self.last_block = block_num"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Found ml-1m Data\n"
     ]
    }
   ],
   "source": [
    "data_dir = './'\n",
    "download_extract('ml-1m', data_dir)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 先来看看数据"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本项目使用的是MovieLens 1M 数据集，包含6000个用户在近4000部电影上的1亿条评论。\n",
    "\n",
    "数据集分为三个文件：用户数据users.dat，电影数据movies.dat和评分数据ratings.dat。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 用户数据\n",
    "分别有用户ID、性别、年龄、职业ID和邮编等字段。\n",
    "\n",
    "数据中的格式：UserID::Gender::Age::Occupation::Zip-code\n",
    "\n",
    "- Gender is denoted by a \"M\" for male and \"F\" for female\n",
    "- Age is chosen from the following ranges:\n",
    "\n",
    "\t*  1:  \"Under 18\"\n",
    "\t* 18:  \"18-24\"\n",
    "\t* 25:  \"25-34\"\n",
    "\t* 35:  \"35-44\"\n",
    "\t* 45:  \"45-49\"\n",
    "\t* 50:  \"50-55\"\n",
    "\t* 56:  \"56+\"\n",
    "\n",
    "- Occupation is chosen from the following choices:\n",
    "\n",
    "\t*  0:  \"other\" or not specified\n",
    "\t*  1:  \"academic/educator\"\n",
    "\t*  2:  \"artist\"\n",
    "\t*  3:  \"clerical/admin\"\n",
    "\t*  4:  \"college/grad student\"\n",
    "\t*  5:  \"customer service\"\n",
    "\t*  6:  \"doctor/health care\"\n",
    "\t*  7:  \"executive/managerial\"\n",
    "\t*  8:  \"farmer\"\n",
    "\t*  9:  \"homemaker\"\n",
    "\t* 10:  \"K-12 student\"\n",
    "\t* 11:  \"lawyer\"\n",
    "\t* 12:  \"programmer\"\n",
    "\t* 13:  \"retired\"\n",
    "\t* 14:  \"sales/marketing\"\n",
    "\t* 15:  \"scientist\"\n",
    "\t* 16:  \"self-employed\"\n",
    "\t* 17:  \"technician/engineer\"\n",
    "\t* 18:  \"tradesman/craftsman\"\n",
    "\t* 19:  \"unemployed\"\n",
    "\t* 20:  \"writer\"\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>OccupationID</th>\n",
       "      <th>Zip-code</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>F</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>48067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>M</td>\n",
       "      <td>56</td>\n",
       "      <td>16</td>\n",
       "      <td>70072</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>15</td>\n",
       "      <td>55117</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>M</td>\n",
       "      <td>45</td>\n",
       "      <td>7</td>\n",
       "      <td>02460</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>M</td>\n",
       "      <td>25</td>\n",
       "      <td>20</td>\n",
       "      <td>55455</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID Gender  Age  OccupationID Zip-code\n",
       "0       1      F    1            10    48067\n",
       "1       2      M   56            16    70072\n",
       "2       3      M   25            15    55117\n",
       "3       4      M   45             7    02460\n",
       "4       5      M   25            20    55455"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users_title = ['UserID', 'Gender', 'Age', 'OccupationID', 'Zip-code']\n",
    "users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "users.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看出UserID、Gender、Age和Occupation都是类别字段，其中邮编字段是我们不使用的。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 电影数据\n",
    "分别有电影ID、电影名和电影风格等字段。\n",
    "\n",
    "数据中的格式：MovieID::Title::Genres\n",
    "\n",
    "- Titles are identical to titles provided by the IMDB (including\n",
    "year of release)\n",
    "- Genres are pipe-separated and are selected from the following genres:\n",
    "\n",
    "\t* Action\n",
    "\t* Adventure\n",
    "\t* Animation\n",
    "\t* Children's\n",
    "\t* Comedy\n",
    "\t* Crime\n",
    "\t* Documentary\n",
    "\t* Drama\n",
    "\t* Fantasy\n",
    "\t* Film-Noir\n",
    "\t* Horror\n",
    "\t* Musical\n",
    "\t* Mystery\n",
    "\t* Romance\n",
    "\t* Sci-Fi\n",
    "\t* Thriller\n",
    "\t* War\n",
    "\t* Western\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>Toy Story (1995)</td>\n",
       "      <td>Animation|Children's|Comedy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>Jumanji (1995)</td>\n",
       "      <td>Adventure|Children's|Fantasy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>Grumpier Old Men (1995)</td>\n",
       "      <td>Comedy|Romance</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>Waiting to Exhale (1995)</td>\n",
       "      <td>Comedy|Drama</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>Father of the Bride Part II (1995)</td>\n",
       "      <td>Comedy</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                               Title                        Genres\n",
       "0        1                    Toy Story (1995)   Animation|Children's|Comedy\n",
       "1        2                      Jumanji (1995)  Adventure|Children's|Fantasy\n",
       "2        3             Grumpier Old Men (1995)                Comedy|Romance\n",
       "3        4            Waiting to Exhale (1995)                  Comedy|Drama\n",
       "4        5  Father of the Bride Part II (1995)                        Comedy"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies_title = ['MovieID', 'Title', 'Genres']\n",
    "movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "movies.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MovieID是类别字段，Title是文本，Genres也是类别字段"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 评分数据\n",
    "分别有用户ID、电影ID、评分和时间戳等字段。\n",
    "\n",
    "数据中的格式：UserID::MovieID::Rating::Timestamp\n",
    "\n",
    "- UserIDs range between 1 and 6040 \n",
    "- MovieIDs range between 1 and 3952\n",
    "- Ratings are made on a 5-star scale (whole-star ratings only)\n",
    "- Timestamp is represented in seconds since the epoch as returned by time(2)\n",
    "- Each user has at least 20 ratings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Rating</th>\n",
       "      <th>timestamps</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1193</td>\n",
       "      <td>5</td>\n",
       "      <td>978300760</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>661</td>\n",
       "      <td>3</td>\n",
       "      <td>978302109</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>914</td>\n",
       "      <td>3</td>\n",
       "      <td>978301968</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>3408</td>\n",
       "      <td>4</td>\n",
       "      <td>978300275</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>2355</td>\n",
       "      <td>5</td>\n",
       "      <td>978824291</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  MovieID  Rating  timestamps\n",
       "0       1     1193       5   978300760\n",
       "1       1      661       3   978302109\n",
       "2       1      914       3   978301968\n",
       "3       1     3408       4   978300275\n",
       "4       1     2355       5   978824291"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings_title = ['UserID','MovieID', 'Rating', 'timestamps']\n",
    "ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "ratings.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "评分字段Rating就是我们要学习的targets，时间戳字段我们不使用。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 来说说数据预处理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- UserID、Occupation和MovieID不用变。\n",
    "- Gender字段：需要将‘F’和‘M’转换成0和1。\n",
    "- Age字段：要转成7个连续数字0~6。\n",
    "- Genres字段：是分类字段，要转成数字。首先将Genres中的类别转成字符串到数字的字典，然后再将每个电影的Genres字段转成数字列表，因为有些电影是多个Genres的组合。\n",
    "- Title字段：处理方式跟Genres字段一样，首先创建文本到数字的字典，然后将Title中的描述转成数字的列表。另外Title中的年份也需要去掉。\n",
    "- Genres和Title字段需要将长度统一，这样在神经网络中方便处理。空白部分用‘< PAD >’对应的数字填充。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 实现数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    \"\"\"\n",
    "    Load Dataset from File\n",
    "    \"\"\"\n",
    "    #读取User数据\n",
    "    users_title = ['UserID', 'Gender', 'Age', 'JobID', 'Zip-code']\n",
    "    users = pd.read_table('./ml-1m/users.dat', sep='::', header=None, names=users_title, engine = 'python')\n",
    "    # 保留以下特征\n",
    "    users = users.filter(regex='UserID|Gender|Age|JobID')\n",
    "    users_orig = users.values\n",
    "    #改变User数据中性别和年龄\n",
    "    gender_map = {'F':0, 'M':1}\n",
    "    users['Gender'] = users['Gender'].map(gender_map)\n",
    "\n",
    "    age_map = {val:ii for ii,val in enumerate(set(users['Age']))}\n",
    "    users['Age'] = users['Age'].map(age_map)\n",
    "\n",
    "    #读取Movie数据集\n",
    "    movies_title = ['MovieID', 'Title', 'Genres']\n",
    "    movies = pd.read_table('./ml-1m/movies.dat', sep='::', header=None, names=movies_title, engine = 'python')\n",
    "    movies_orig = movies.values\n",
    "    #将Title中的年份去掉\n",
    "    pattern = re.compile(r'^(.*)\\((\\d+)\\)$')\n",
    "\n",
    "    title_map = {val:pattern.match(val).group(1) for ii,val in enumerate(set(movies['Title']))}\n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #电影类型转数字字典\n",
    "    genres_set = set()\n",
    "    for val in movies['Genres'].str.split('|'):\n",
    "        genres_set.update(val)\n",
    "\n",
    "    genres_set.add('<PAD>')\n",
    "    # 将类型进行编号\n",
    "    genres2int = {val:ii for ii, val in enumerate(genres_set)}\n",
    "\n",
    "    #将电影类型转成等长数字列表\n",
    "    genres_map = {val:[genres2int[row] for row in val.split('|')] for ii,val in enumerate(set(movies['Genres']))}\n",
    "\n",
    "    # 将每个样本的电影类型数字列表处理成相同长度，长度不够用<PAD>填充\n",
    "    for key in genres_map:\n",
    "        for cnt in range(max(genres2int.values()) - len(genres_map[key])):\n",
    "            genres_map[key].insert(len(genres_map[key]) + cnt,genres2int['<PAD>'])\n",
    "    \n",
    "    movies['Genres'] = movies['Genres'].map(genres_map)\n",
    "\n",
    "    #电影Title转数字字典\n",
    "    title_set = set()\n",
    "    for val in movies['Title'].str.split():\n",
    "        title_set.update(val)\n",
    "    \n",
    "    title_set.add('<PAD>')\n",
    "    title2int = {val:ii for ii, val in enumerate(title_set)}\n",
    "\n",
    "    #将电影Title转成等长数字列表，长度是15\n",
    "    title_count = 15\n",
    "    title_map = {val:[title2int[row] for row in val.split()] for ii,val in enumerate(set(movies['Title']))}\n",
    "    \n",
    "    for key in title_map:\n",
    "        for cnt in range(title_count - len(title_map[key])):\n",
    "            title_map[key].insert(len(title_map[key]) + cnt,title2int['<PAD>'])\n",
    "    \n",
    "    movies['Title'] = movies['Title'].map(title_map)\n",
    "\n",
    "    #读取评分数据集\n",
    "    ratings_title = ['UserID','MovieID', 'ratings', 'timestamps']\n",
    "    ratings = pd.read_table('./ml-1m/ratings.dat', sep='::', header=None, names=ratings_title, engine = 'python')\n",
    "    ratings = ratings.filter(regex='UserID|MovieID|ratings')\n",
    "\n",
    "    #合并三个表\n",
    "    data = pd.merge(pd.merge(ratings, users), movies)\n",
    "    \n",
    "    #将数据分成X和y两张表\n",
    "    target_fields = ['ratings']\n",
    "    features_pd, targets_pd = data.drop(target_fields, axis=1), data[target_fields]\n",
    "    \n",
    "    features = features_pd.values\n",
    "    targets_values = targets_pd.values\n",
    "    \n",
    "    return title_count, title_set, genres2int, features, targets_values, ratings, users, movies, data, movies_orig, users_orig"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 加载数据并保存到本地"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- title_count：Title字段的长度（15）\n",
    "- title_set：Title文本的集合\n",
    "- genres2int：电影类型转数字的字典\n",
    "- features：是输入X\n",
    "- targets_values：是学习目标y\n",
    "- ratings：评分数据集的Pandas对象\n",
    "- users：用户数据集的Pandas对象\n",
    "- movies：电影数据的Pandas对象\n",
    "- data：三个数据集组合在一起的Pandas对象\n",
    "- movies_orig：没有做数据处理的原始电影数据\n",
    "- users_orig：没有做数据处理的原始用户数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "title_count, title_set, genres2int, features, targets_values, \\\n",
    "        ratings, users, movies, data, movies_orig, users_orig = load_data()\n",
    "\n",
    "with open('./processed_data/preprocess.pkl', 'wb') as f:\n",
    "    pickle.dump((title_count, \n",
    "                 title_set, \n",
    "                 genres2int, \n",
    "                 features, \n",
    "                 targets_values, \n",
    "                 ratings, \n",
    "                 users, \n",
    "                 movies, \n",
    "                 data, \n",
    "                 movies_orig, \n",
    "                 users_orig), \n",
    "                f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预处理后的数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserID</th>\n",
       "      <th>Gender</th>\n",
       "      <th>Age</th>\n",
       "      <th>JobID</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>16</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>20</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   UserID  Gender  Age  JobID\n",
       "0       1       0    0     10\n",
       "1       2       1    5     16\n",
       "2       3       1    6     15\n",
       "3       4       1    2      7\n",
       "4       5       1    6     20"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "users.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>MovieID</th>\n",
       "      <th>Title</th>\n",
       "      <th>Genres</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>[2770, 1146, 4239, 4239, 4239, 4239, 4239, 423...</td>\n",
       "      <td>[13, 10, 15, 18, 18, 18, 18, 18, 18, 18, 18, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>[4608, 4239, 4239, 4239, 4239, 4239, 4239, 423...</td>\n",
       "      <td>[5, 10, 2, 18, 18, 18, 18, 18, 18, 18, 18, 18,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>[1812, 3287, 3404, 4239, 4239, 4239, 4239, 423...</td>\n",
       "      <td>[15, 11, 18, 18, 18, 18, 18, 18, 18, 18, 18, 1...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>[1702, 359, 4435, 4239, 4239, 4239, 4239, 4239...</td>\n",
       "      <td>[15, 7, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>[4146, 3163, 2691, 4448, 2800, 677, 4239, 4239...</td>\n",
       "      <td>[15, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 1...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   MovieID                                              Title  \\\n",
       "0        1  [2770, 1146, 4239, 4239, 4239, 4239, 4239, 423...   \n",
       "1        2  [4608, 4239, 4239, 4239, 4239, 4239, 4239, 423...   \n",
       "2        3  [1812, 3287, 3404, 4239, 4239, 4239, 4239, 423...   \n",
       "3        4  [1702, 359, 4435, 4239, 4239, 4239, 4239, 4239...   \n",
       "4        5  [4146, 3163, 2691, 4448, 2800, 677, 4239, 4239...   \n",
       "\n",
       "                                              Genres  \n",
       "0  [13, 10, 15, 18, 18, 18, 18, 18, 18, 18, 18, 1...  \n",
       "1  [5, 10, 2, 18, 18, 18, 18, 18, 18, 18, 18, 18,...  \n",
       "2  [15, 11, 18, 18, 18, 18, 18, 18, 18, 18, 18, 1...  \n",
       "3  [15, 7, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18...  \n",
       "4  [15, 18, 18, 18, 18, 18, 18, 18, 18, 18, 18, 1...  "
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "movies.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000209, 7)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 从本地读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./processed_data/preprocess.pkl', mode='rb') as f:\n",
    "    title_count, title_set, genres2int, features, \\\n",
    "        targets_values, ratings, users, movies, data, movies_orig, users_orig = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型设计"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/model.001.jpeg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "通过研究数据集中的字段类型，我们发现有一些是类别字段，通常的处理是将这些字段转成one hot编码，但是像UserID、MovieID这样的字段就会变成非常的稀疏，输入的维度急剧膨胀，这是我们不愿意见到的，毕竟我这小笔记本不像大厂动辄能处理数以亿计维度的输入：）\n",
    "\n",
    "所以在预处理数据时将这些字段转成了数字，我们用这个数字当做嵌入矩阵的索引，在网络的第一层使用了嵌入层，维度是（N，32）和（N，16）。\n",
    "这里的思想其实和 word2vec 比较类似。我们会对用户或者电影的每个属性都指定一个特征维度空间，这就好比我们在自然语言处理中对每个单词指定特征维度空间。从下面的代码中可以看到，我们将用到的属性的特征维度设置为了 32 或者 16.\n",
    "\n",
    "电影类型的处理要多一步，有时一个电影有多个电影类型，这样从嵌入矩阵索引出来是一个（n，32）的矩阵，因为有多个类型嘛，我们要将这个矩阵求和，变成（1，32）的向量。\n",
    "\n",
    "电影名的处理比较特殊，没有使用循环神经网络，而是用了文本卷积网络，下文会进行说明。\n",
    "\n",
    "从嵌入层索引出特征以后，将各特征传入全连接层，将输出再次传入全连接层，最终分别得到（1，200）的用户特征和电影特征两个特征向量。\n",
    "\n",
    "我们的目的就是要训练出用户特征和电影特征，在实现推荐功能时使用。得到这两个特征以后，就可以选择任意的方式来拟合评分了。我使用了两种方式，一个是上图中画出的将两个特征做向量乘法，将结果与真实评分做回归，采用MSE优化损失。因为本质上这是一个回归问题，另一种方式是，将两个特征作为输入，再次传入全连接层，输出一个值，将输出值回归到真实评分，采用MSE优化损失。\n",
    "\n",
    "实际上第二个方式的MSE loss在0.8附近，第一个方式在1附近，5次迭代的结果。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 文本卷积网络\n",
    "网络看起来像下面这样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"assets/text_cnn.png\"/>\n",
    "图片来自Kim Yoon的论文：[`Convolutional Neural Networks for Sentence Classification`](https://arxiv.org/abs/1408.5882)\n",
    "\n",
    "将卷积神经网络用于文本的文章建议你阅读[`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "网络的第一层是词嵌入层，由每一个单词的嵌入向量组成的嵌入矩阵。下一层使用多个不同尺寸（窗口大小）的卷积核在嵌入矩阵上做卷积，<b>窗口大小指的是每次卷积覆盖几个单词</b>。这里跟对图像做卷积不太一样，图像的卷积通常用2x2、3x3、5x5之类的尺寸，而文本卷积要覆盖整个单词的嵌入向量，所以尺寸是（单词数，向量维度），比如每次滑动3个，4个或者5个单词。第三层网络是max pooling得到一个长向量，最后使用dropout做正则化，最终得到了电影Title的特征。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 辅助函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import os\n",
    "import pickle\n",
    "\n",
    "def save_params(params):\n",
    "    \"\"\"\n",
    "    Save parameters to file\n",
    "    \"\"\"\n",
    "    with open('params.p', 'wb') as f:\n",
    "        pickle.dump(params, f)\n",
    "\n",
    "\n",
    "def load_params():\n",
    "    \"\"\"\n",
    "    Load parameters from file\n",
    "    \"\"\"\n",
    "    with open('params.p', mode='rb') as f:\n",
    "        return pickle.load(f)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 编码实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "#嵌入矩阵的维度\n",
    "embed_dim = 32\n",
    "# 下面之所以要 +1 是因为编号和实际数量之间是差 1 的\n",
    "#用户ID个数\n",
    "uid_max = max(features.take(0,1)) + 1 # 6040\n",
    "#性别个数\n",
    "gender_max = max(features.take(2,1)) + 1 # 1 + 1 = 2\n",
    "#年龄类别个数\n",
    "age_max = max(features.take(3,1)) + 1 # 6 + 1 = 7\n",
    "#职业个数\n",
    "job_max = max(features.take(4,1)) + 1# 20 + 1 = 21\n",
    "\n",
    "#电影ID个数\n",
    "movie_id_max = max(features.take(1,1)) + 1 # 3952\n",
    "#电影类型个数，有个<PAD>\n",
    "movie_categories_max = max(genres2int.values()) + 1 # 18 + 1 = 19\n",
    "#电影名单词个数\n",
    "movie_title_max = len(title_set) # 5216\n",
    "\n",
    "#对电影类型嵌入向量做加和操作的标志，考虑过使用mean做平均，但是没实现mean\n",
    "combiner = \"sum\"\n",
    "\n",
    "#电影名长度，做词嵌入要求输入的维度是固定的，这里设置为 15\n",
    "# 长度不够用空白符填充，太长则进行截断\n",
    "sentences_size = title_count # = 15\n",
    "#文本卷积滑动窗口，分别滑动2, 3, 4, 5个单词\n",
    "window_sizes = {2, 3, 4, 5}\n",
    "#文本卷积核数量\n",
    "filter_num = 8\n",
    "\n",
    "#电影ID转下标的字典，数据集中电影ID跟下标不一致，比如第5行的数据电影ID不一定是5\n",
    "movieid2idx = {val[0]:i for i, val in enumerate(movies.values)}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 超参"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of Epochs\n",
    "num_epochs = 5\n",
    "# Batch Size\n",
    "batch_size = 256\n",
    "\n",
    "dropout_keep = 0.5\n",
    "# Learning Rate\n",
    "learning_rate = 0.0001\n",
    "# Show stats for every n number of batches\n",
    "show_every_n_batches = 50\n",
    "\n",
    "save_dir = './save'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 输入"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义输入的占位符"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_inputs():\n",
    "    uid = tf.placeholder(tf.int32, [None, 1], name=\"uid\")\n",
    "    user_gender = tf.placeholder(tf.int32, [None, 1], name=\"user_gender\")\n",
    "    user_age = tf.placeholder(tf.int32, [None, 1], name=\"user_age\")\n",
    "    user_job = tf.placeholder(tf.int32, [None, 1], name=\"user_job\")\n",
    "    \n",
    "    movie_id = tf.placeholder(tf.int32, [None, 1], name=\"movie_id\")\n",
    "    # 电影种类中要去除<PAD>，所以-1\n",
    "    movie_categories = tf.placeholder(tf.int32, [None, movie_categories_max-1], name=\"movie_categories\")\n",
    "    movie_titles = tf.placeholder(tf.int32, [None, 15], name=\"movie_titles\")\n",
    "    targets = tf.placeholder(tf.int32, [None, 1], name=\"targets\")\n",
    "    LearningRate = tf.placeholder(tf.float32, name = \"LearningRate\")\n",
    "    dropout_keep_prob = tf.placeholder(tf.float32, name = \"dropout_keep_prob\")\n",
    "    return uid, user_gender, user_age, user_job, \\\n",
    "            movie_id, movie_categories, movie_titles, targets, \\\n",
    "                LearningRate, dropout_keep_prob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义User的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 从上面的网络架构图中，我们可以看出，其实用户的特征总数量是 128\n",
    "# 但是作者在下面的代码中并没有这样做，而是由的特征设置为了 16，这样可能是因为作者的电脑性能比较差\n",
    "# 我们先尝试这样的设置，如果计算资源允许，我们在后面再次测试的时候全部设置为 32\n",
    "def get_user_embedding(uid, user_gender, user_age, user_job):\n",
    "    with tf.name_scope(\"user_embedding\"):\n",
    "        # 下面的操作和情感分析项目中的单词转换为词向量的操作本质上是一样的\n",
    "        # 用户的特征维度设置为 32\n",
    "        # 先初始化一个非常大的用户矩阵\n",
    "        # tf.random_uniform 的第二个参数是初始化的最小值，这里是-1，第三个参数是初始化的最大值，这里是1\n",
    "        uid_embed_matrix = tf.Variable(tf.random_uniform([uid_max, embed_dim], -1, 1), \n",
    "                                       name = \"uid_embed_matrix\")\n",
    "        # 根据指定用户ID找到他对应的嵌入层\n",
    "        uid_embed_layer = tf.nn.embedding_lookup(uid_embed_matrix, uid, \n",
    "                                                 name = \"uid_embed_layer\")\n",
    "    \n",
    "        # 性别的特征维度设置为 16\n",
    "#         gender_embed_matrix = tf.Variable(tf.random_uniform([gender_max, embed_dim // 2], -1, 1), \n",
    "#                                           name= \"gender_embed_matrix\")\n",
    "        gender_embed_matrix = tf.Variable(tf.random_uniform([gender_max, embed_dim], -1, 1), \n",
    "                                          name= \"gender_embed_matrix\")\n",
    "        gender_embed_layer = tf.nn.embedding_lookup(gender_embed_matrix, user_gender, \n",
    "                                                    name = \"gender_embed_layer\")\n",
    "        \n",
    "        # 年龄的特征维度设置为 16\n",
    "#         age_embed_matrix = tf.Variable(tf.random_uniform([age_max, embed_dim // 2], -1, 1),\n",
    "#                                        name=\"age_embed_matrix\")\n",
    "        age_embed_matrix = tf.Variable(tf.random_uniform([age_max, embed_dim], -1, 1),\n",
    "                                       name=\"age_embed_matrix\")\n",
    "        age_embed_layer = tf.nn.embedding_lookup(age_embed_matrix, user_age, \n",
    "                                                 name=\"age_embed_layer\")\n",
    "        \n",
    "        # 职业的特征维度设置为 16\n",
    "#         job_embed_matrix = tf.Variable(tf.random_uniform([job_max, embed_dim // 2], -1, 1), \n",
    "#                                        name = \"job_embed_matrix\")\n",
    "        job_embed_matrix = tf.Variable(tf.random_uniform([job_max, embed_dim], -1, 1), \n",
    "                                       name = \"job_embed_matrix\")\n",
    "        job_embed_layer = tf.nn.embedding_lookup(job_embed_matrix, user_job, \n",
    "                                                 name = \"job_embed_layer\")\n",
    "    # 返回产生的用户数据数据\n",
    "    return uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将User的嵌入矩阵一起全连接生成User的特征"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer):\n",
    "    with tf.name_scope(\"user_fc\"):\n",
    "        #第一层全连接\n",
    "        # tf.layers.dense 的第一个参数是输入，第二个参数是层的单元的数量\n",
    "        uid_fc_layer = tf.layers.dense(uid_embed_layer, embed_dim, name = \"uid_fc_layer\", activation=tf.nn.relu)\n",
    "        gender_fc_layer = tf.layers.dense(gender_embed_layer, embed_dim, name = \"gender_fc_layer\", activation=tf.nn.relu)\n",
    "        age_fc_layer = tf.layers.dense(age_embed_layer, embed_dim, name =\"age_fc_layer\", activation=tf.nn.relu)\n",
    "        job_fc_layer = tf.layers.dense(job_embed_layer, embed_dim, name = \"job_fc_layer\", activation=tf.nn.relu)\n",
    "        \n",
    "        #第二层全连接\n",
    "        # 将上面的每个分段组成一个完整的全连接层\n",
    "        user_combine_layer = tf.concat([uid_fc_layer, gender_fc_layer, age_fc_layer, job_fc_layer], 2)  #(?, 1, 128)\n",
    "        # 验证上面产生的 tensorflow 是否是 128 维度的\n",
    "        print(user_combine_layer.shape)\n",
    "        # tf.contrib.layers.fully_connected 的第一个参数是输入，第二个参数是输出\n",
    "        # 这里的输入是user_combine_layer，输出是200，是指每个用户有200个特征\n",
    "        # 相当于是一个200个分类的问题，每个分类的可能性都会输出，在这里指的就是每个特征的可能性\n",
    "        user_combine_layer = tf.contrib.layers.fully_connected(user_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        user_combine_layer_flat = tf.reshape(user_combine_layer, [-1, 200])\n",
    "    return user_combine_layer, user_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 定义Movie ID的嵌入矩阵"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_id_embed_layer(movie_id):\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_id_embed_matrix = tf.Variable(tf.random_uniform([movie_id_max, embed_dim], -1, 1), name = \"movie_id_embed_matrix\")\n",
    "        movie_id_embed_layer = tf.nn.embedding_lookup(movie_id_embed_matrix, movie_id, name = \"movie_id_embed_layer\")\n",
    "    return movie_id_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 对电影类型的多个嵌入向量做加和"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_categories_layers(movie_categories):\n",
    "    with tf.name_scope(\"movie_categories_layers\"):\n",
    "        movie_categories_embed_matrix = tf.Variable(tf.random_uniform([movie_categories_max, embed_dim], -1, 1), name = \"movie_categories_embed_matrix\")\n",
    "        movie_categories_embed_layer = tf.nn.embedding_lookup(movie_categories_embed_matrix, movie_categories, name = \"movie_categories_embed_layer\")\n",
    "        if combiner == \"sum\":\n",
    "            movie_categories_embed_layer = tf.reduce_sum(movie_categories_embed_layer, axis=1, keep_dims=True)\n",
    "    #     elif combiner == \"mean\":\n",
    "\n",
    "    return movie_categories_embed_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movie Title的文本卷积网络实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_cnn_layer(movie_titles):\n",
    "    #从嵌入矩阵中得到电影名对应的各个单词的嵌入向量\n",
    "    with tf.name_scope(\"movie_embedding\"):\n",
    "        movie_title_embed_matrix = tf.Variable(tf.random_uniform([movie_title_max, embed_dim], -1, 1), name = \"movie_title_embed_matrix\")\n",
    "        movie_title_embed_layer = tf.nn.embedding_lookup(movie_title_embed_matrix, movie_titles, name = \"movie_title_embed_layer\")\n",
    "        # 为 movie_title_embed_layer 增加一个维度\n",
    "        # 在这里是添加到最后一个维度，最后一个维度是channel\n",
    "        # 所以这里的channel数量是1个\n",
    "        # 所以这里的处理方式和图片是一样的\n",
    "        movie_title_embed_layer_expand = tf.expand_dims(movie_title_embed_layer, -1)\n",
    "    \n",
    "    #对文本嵌入层使用不同尺寸的卷积核做卷积和最大池化\n",
    "    pool_layer_lst = []\n",
    "    for window_size in window_sizes:\n",
    "        with tf.name_scope(\"movie_txt_conv_maxpool_{}\".format(window_size)):\n",
    "            # [window_size, embed_dim, 1, filter_num] 表示输入的 channel 的个数是1，输出的 channel 的个数是 filter_num\n",
    "            filter_weights = tf.Variable(tf.truncated_normal([window_size, embed_dim, 1, filter_num],stddev=0.1),name = \"filter_weights\")\n",
    "            filter_bias = tf.Variable(tf.constant(0.1, shape=[filter_num]), name=\"filter_bias\")\n",
    "            \n",
    "            # conv2d 是指用到的卷积核的大小是 [filter_height * filter_width * in_channels, output_channels]\n",
    "            # 在这里卷积核会向两个维度的方向进行滑动\n",
    "            # conv1d 是将卷积核向一个维度的方向进行滑动，这就是 conv1d 和 conv2d 的区别\n",
    "            # strides 设置要求第一个和最后一个数字是1，四个数字的顺序要求默认是 NHWC，也就是 [batch, height, width, channels]\n",
    "            # padding 设置为 VALID 其实就是不 PAD，设置为 SAME 就是让输入和输出的维度是一样的\n",
    "            conv_layer = tf.nn.conv2d(movie_title_embed_layer_expand, filter_weights, [1,1,1,1], padding=\"VALID\", name=\"conv_layer\")\n",
    "            # tf.nn.bias_add 将偏差 filter_bias 加到 conv_layer 上\n",
    "            # tf.nn.relu 将激活函数设置为 relu\n",
    "            relu_layer = tf.nn.relu(tf.nn.bias_add(conv_layer,filter_bias), name =\"relu_layer\")\n",
    "            \n",
    "            \n",
    "            # tf.nn.max_pool 的第一个参数是输入\n",
    "            # 第二个参数是 max_pool 窗口的大小，每个数值表示对每个维度的窗口设置\n",
    "            # 第三个参数是 strides，和 conv2d 的设置是一样的\n",
    "            # 这边的池化是将上面每个卷积核的卷积结果转换为一个元素\n",
    "            # 由于这里的卷积核的数量是 8 个，所以下面生成的是一个具有 8 个元素的向量\n",
    "            maxpool_layer = tf.nn.max_pool(relu_layer, [1,sentences_size - window_size + 1 ,1,1], [1,1,1,1], padding=\"VALID\", name=\"maxpool_layer\")\n",
    "            pool_layer_lst.append(maxpool_layer)\n",
    "\n",
    "    #Dropout层\n",
    "    with tf.name_scope(\"pool_dropout\"):\n",
    "        # 这里最终的结果是这样的，\n",
    "        # 假设卷积核的窗口是 2，卷积核的数量是 8\n",
    "        # 那么通过上面的池化操作之后，生成的池化的结果是一个具有 8 个元素的向量\n",
    "        # 每种窗口大小的卷积核经过池化后都会生成这样一个具有 8 个元素的向量\n",
    "        # 所以最终生成的是一个 8 维的二维矩阵，它的另一个维度就是不同的窗口的数量\n",
    "        # 在这里就是 2,3,4,5，那么最终就是一个 8*4 的矩阵，\n",
    "        pool_layer = tf.concat(pool_layer_lst, 3, name =\"pool_layer\")\n",
    "        max_num = len(window_sizes) * filter_num\n",
    "        # 将这个 8*4 的二维矩阵平铺成一个具有 32 个元素的一维矩阵\n",
    "        pool_layer_flat = tf.reshape(pool_layer , [-1, 1, max_num], name = \"pool_layer_flat\")\n",
    "    \n",
    "        dropout_layer = tf.nn.dropout(pool_layer_flat, dropout_keep_prob, name = \"dropout_layer\")\n",
    "    return pool_layer_flat, dropout_layer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 将Movie的各个层一起做全连接"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_movie_feature_layer(movie_id_embed_layer, movie_categories_embed_layer, dropout_layer):\n",
    "    with tf.name_scope(\"movie_fc\"):\n",
    "        #第一层全连接\n",
    "        movie_id_fc_layer = tf.layers.dense(movie_id_embed_layer, embed_dim, name = \"movie_id_fc_layer\", activation=tf.nn.relu)\n",
    "        movie_categories_fc_layer = tf.layers.dense(movie_categories_embed_layer, embed_dim, name = \"movie_categories_fc_layer\", activation=tf.nn.relu)\n",
    "    \n",
    "        #第二层全连接\n",
    "        movie_combine_layer = tf.concat([movie_id_fc_layer, movie_categories_fc_layer, dropout_layer], 2)  #(?, 1, 96)\n",
    "        movie_combine_layer = tf.contrib.layers.fully_connected(movie_combine_layer, 200, tf.tanh)  #(?, 1, 200)\n",
    "    \n",
    "        movie_combine_layer_flat = tf.reshape(movie_combine_layer, [-1, 200])\n",
    "    return movie_combine_layer, movie_combine_layer_flat"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建计算图"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(?, 1, 128)\n",
      "WARNING:tensorflow:From <ipython-input-20-559a1ee9ce9e>:6: calling reduce_sum (from tensorflow.python.ops.math_ops) with keep_dims is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "keep_dims is deprecated, use keepdims instead\n"
     ]
    }
   ],
   "source": [
    "# reset_default_graph 操作应该在 tensorflow 的其他所有操作之前进行，否则将会出现不可知的问题\n",
    "# tensorflow 中的 graph 包含的是一系列的操作和使用到这些操作的 tensor\n",
    "tf.reset_default_graph()\n",
    "train_graph = tf.Graph()\n",
    "# Graph 只是当前线程的属性，如果想要在其他的线程使用这个 Graph，那么就要像下面这样指定\n",
    "# 下面的 with 语句将 train_graph 设置为当前线程的默认 graph\n",
    "with train_graph.as_default():\n",
    "    #获取输入占位符\n",
    "    uid, user_gender, user_age, user_job, \\\n",
    "        movie_id, movie_categories, movie_titles, \\\n",
    "            targets, lr, dropout_keep_prob = get_inputs()\n",
    "    #获取User的4个嵌入向量\n",
    "    uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer = get_user_embedding(uid, user_gender, user_age, user_job)\n",
    "    #得到用户特征\n",
    "    user_combine_layer, user_combine_layer_flat = get_user_feature_layer(uid_embed_layer, gender_embed_layer, age_embed_layer, job_embed_layer)\n",
    "    #获取电影ID的嵌入向量\n",
    "    movie_id_embed_layer = get_movie_id_embed_layer(movie_id)\n",
    "    #获取电影类型的嵌入向量\n",
    "    movie_categories_embed_layer = get_movie_categories_layers(movie_categories)\n",
    "    #获取电影名的特征向量\n",
    "    pool_layer_flat, dropout_layer = get_movie_cnn_layer(movie_titles)\n",
    "    #得到电影特征\n",
    "    movie_combine_layer, movie_combine_layer_flat = get_movie_feature_layer(movie_id_embed_layer, \n",
    "                                                                                movie_categories_embed_layer, \n",
    "                                                                                dropout_layer)\n",
    "    #计算出评分，要注意两个不同的方案，inference的名字（name值）是不一样的，后面做推荐时要根据name取得tensor\n",
    "    # tensorflow 的 name_scope 指定了 tensor 范围，方便我们后面调用，通过指定 name_scope 来调用其中的 tensor\n",
    "    with tf.name_scope(\"inference\"):\n",
    "        # 直接将用户特征矩阵和电影特征矩阵相乘得到得分，最后要做的就是对这个得分进行回归\n",
    "        inference = tf.reduce_sum(user_combine_layer_flat * movie_combine_layer_flat, axis=1)\n",
    "        inference = tf.expand_dims(inference, axis=1)\n",
    "\n",
    "    with tf.name_scope(\"loss\"):\n",
    "        # MSE损失，将计算值回归到评分\n",
    "        cost = tf.losses.mean_squared_error(targets, inference )\n",
    "        # 将每个维度的 cost 相加，计算它们的平均值\n",
    "        loss = tf.reduce_mean(cost)\n",
    "    # 优化损失 \n",
    "#     train_op = tf.train.AdamOptimizer(lr).minimize(loss)  #cost\n",
    "# 在为 tensorflow 设置 name 参数的时候，是为了能在 graph 中看到什么变量进行了什么操作\n",
    "    global_step = tf.Variable(0, name=\"global_step\", trainable=False)\n",
    "#     optimizer = tf.train.AdamOptimizer(lr)\n",
    "    optimizer = tf.train.AdamOptimizer()\n",
    "    gradients = optimizer.compute_gradients(loss)  #cost\n",
    "    train_op = optimizer.apply_gradients(gradients, global_step=global_step)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 取得batch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 自定义获取 batch 的方法\n",
    "def get_batches(Xs, ys, batch_size):\n",
    "    for start in range(0, len(Xs), batch_size):\n",
    "        end = min(start + batch_size, len(Xs))\n",
    "        yield Xs[start:end], ys[start:end]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 训练网络"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing to D:\\28 machine learning\\08 项目\\八维实训项目\\06 电影推荐\\基于CNN的电影推荐系统\\runs\\1536908929\n",
      "\n",
      "2018-09-14T15:08:57.935265: Epoch   0 Batch    0/3125   train_loss = 25.717\n",
      "2018-09-14T15:09:03.961826: Epoch   0 Batch   50/3125   train_loss = 1.420\n",
      "2018-09-14T15:09:09.619593: Epoch   0 Batch  100/3125   train_loss = 1.449\n",
      "2018-09-14T15:09:15.142444: Epoch   0 Batch  150/3125   train_loss = 1.450\n",
      "2018-09-14T15:09:20.093619: Epoch   0 Batch  200/3125   train_loss = 1.397\n",
      "2018-09-14T15:09:24.683006: Epoch   0 Batch  250/3125   train_loss = 1.356\n",
      "2018-09-14T15:09:28.609786: Epoch   0 Batch  300/3125   train_loss = 1.377\n",
      "2018-09-14T15:09:32.593514: Epoch   0 Batch  350/3125   train_loss = 1.220\n",
      "2018-09-14T15:09:36.441319: Epoch   0 Batch  400/3125   train_loss = 1.077\n",
      "2018-09-14T15:09:40.309089: Epoch   0 Batch  450/3125   train_loss = 1.218\n",
      "2018-09-14T15:09:45.538115: Epoch   0 Batch  500/3125   train_loss = 0.906\n",
      "2018-09-14T15:09:50.876064: Epoch   0 Batch  550/3125   train_loss = 1.222\n",
      "2018-09-14T15:09:55.671356: Epoch   0 Batch  600/3125   train_loss = 1.244\n",
      "2018-09-14T15:10:00.614509: Epoch   0 Batch  650/3125   train_loss = 1.381\n",
      "2018-09-14T15:10:04.545291: Epoch   0 Batch  700/3125   train_loss = 1.202\n",
      "2018-09-14T15:10:08.451063: Epoch   0 Batch  750/3125   train_loss = 1.136\n",
      "2018-09-14T15:10:11.774147: Epoch   0 Batch  800/3125   train_loss = 1.095\n",
      "2018-09-14T15:10:14.531590: Epoch   0 Batch  850/3125   train_loss = 1.199\n",
      "2018-09-14T15:10:17.070126: Epoch   0 Batch  900/3125   train_loss = 0.977\n",
      "2018-09-14T15:10:19.507750: Epoch   0 Batch  950/3125   train_loss = 1.154\n",
      "2018-09-14T15:10:21.839420: Epoch   0 Batch 1000/3125   train_loss = 1.117\n",
      "2018-09-14T15:10:23.982180: Epoch   0 Batch 1050/3125   train_loss = 1.081\n",
      "2018-09-14T15:10:25.726187: Epoch   0 Batch 1100/3125   train_loss = 1.080\n",
      "2018-09-14T15:10:27.543152: Epoch   0 Batch 1150/3125   train_loss = 1.059\n",
      "2018-09-14T15:10:29.265183: Epoch   0 Batch 1200/3125   train_loss = 1.106\n",
      "2018-09-14T15:10:30.981204: Epoch   0 Batch 1250/3125   train_loss = 1.151\n",
      "2018-09-14T15:10:32.723211: Epoch   0 Batch 1300/3125   train_loss = 0.891\n",
      "2018-09-14T15:10:34.447227: Epoch   0 Batch 1350/3125   train_loss = 0.933\n",
      "2018-09-14T15:10:36.247202: Epoch   0 Batch 1400/3125   train_loss = 1.012\n",
      "2018-09-14T15:10:38.201088: Epoch   0 Batch 1450/3125   train_loss = 1.139\n",
      "2018-09-14T15:10:40.157972: Epoch   0 Batch 1500/3125   train_loss = 0.994\n",
      "2018-09-14T15:10:42.110858: Epoch   0 Batch 1550/3125   train_loss = 1.034\n",
      "2018-09-14T15:10:44.067742: Epoch   0 Batch 1600/3125   train_loss = 0.882\n",
      "2018-09-14T15:10:46.045614: Epoch   0 Batch 1650/3125   train_loss = 0.957\n",
      "2018-09-14T15:10:48.179401: Epoch   0 Batch 1700/3125   train_loss = 0.913\n",
      "2018-09-14T15:10:50.315164: Epoch   0 Batch 1750/3125   train_loss = 0.870\n",
      "2018-09-14T15:10:55.117148: Epoch   0 Batch 1800/3125   train_loss = 0.976\n",
      "2018-09-14T15:11:00.239258: Epoch   0 Batch 1850/3125   train_loss = 0.910\n",
      "2018-09-14T15:11:05.661165: Epoch   0 Batch 1900/3125   train_loss = 0.817\n",
      "2018-09-14T15:11:11.032071: Epoch   0 Batch 1950/3125   train_loss = 0.904\n",
      "2018-09-14T15:11:15.786360: Epoch   0 Batch 2000/3125   train_loss = 1.071\n",
      "2018-09-14T15:11:20.704582: Epoch   0 Batch 2050/3125   train_loss = 0.972\n",
      "2018-09-14T15:11:25.115038: Epoch   0 Batch 2100/3125   train_loss = 0.816\n",
      "2018-09-14T15:11:28.930863: Epoch   0 Batch 2150/3125   train_loss = 0.904\n",
      "2018-09-14T15:11:33.183437: Epoch   0 Batch 2200/3125   train_loss = 0.868\n",
      "2018-09-14T15:11:37.696886: Epoch   0 Batch 2250/3125   train_loss = 0.998\n",
      "2018-09-14T15:11:41.694603: Epoch   0 Batch 2300/3125   train_loss = 0.904\n",
      "2018-09-14T15:11:45.334526: Epoch   0 Batch 2350/3125   train_loss = 0.986\n",
      "2018-09-14T15:11:48.932475: Epoch   0 Batch 2400/3125   train_loss = 0.920\n",
      "2018-09-14T15:11:52.452447: Epoch   0 Batch 2450/3125   train_loss = 0.938\n",
      "2018-09-14T15:11:56.090376: Epoch   0 Batch 2500/3125   train_loss = 0.899\n",
      "2018-09-14T15:11:59.891205: Epoch   0 Batch 2550/3125   train_loss = 0.955\n",
      "2018-09-14T15:12:02.900506: Epoch   0 Batch 2600/3125   train_loss = 0.871\n",
      "2018-09-14T15:12:05.705889: Epoch   0 Batch 2650/3125   train_loss = 1.022\n",
      "2018-09-14T15:12:08.742157: Epoch   0 Batch 2700/3125   train_loss = 0.984\n",
      "2018-09-14T15:12:13.674344: Epoch   0 Batch 2750/3125   train_loss = 1.049\n",
      "2018-09-14T15:12:19.932782: Epoch   0 Batch 2800/3125   train_loss = 1.059\n",
      "2018-09-14T15:12:26.288150: Epoch   0 Batch 2850/3125   train_loss = 0.901\n",
      "2018-09-14T15:12:32.632532: Epoch   0 Batch 2900/3125   train_loss = 0.830\n",
      "2018-09-14T15:12:37.269887: Epoch   0 Batch 2950/3125   train_loss = 0.963\n",
      "2018-09-14T15:12:41.693363: Epoch   0 Batch 3000/3125   train_loss = 0.995\n",
      "2018-09-14T15:12:45.973922: Epoch   0 Batch 3050/3125   train_loss = 0.813\n",
      "2018-09-14T15:12:48.846284: Epoch   0 Batch 3100/3125   train_loss = 1.044\n",
      "2018-09-14T15:12:50.587297: Epoch   0 Batch    0/781   test_loss = 0.865\n",
      "2018-09-14T15:12:52.249344: Epoch   0 Batch   50/781   test_loss = 0.905\n",
      "2018-09-14T15:12:53.564601: Epoch   0 Batch  100/781   test_loss = 1.034\n",
      "2018-09-14T15:12:54.924841: Epoch   0 Batch  150/781   test_loss = 0.921\n",
      "2018-09-14T15:12:56.288040: Epoch   0 Batch  200/781   test_loss = 0.946\n",
      "2018-09-14T15:12:57.635277: Epoch   0 Batch  250/781   test_loss = 0.817\n",
      "2018-09-14T15:12:59.048466: Epoch   0 Batch  300/781   test_loss = 0.868\n",
      "2018-09-14T15:13:00.377712: Epoch   0 Batch  350/781   test_loss = 0.898\n",
      "2018-09-14T15:13:01.776913: Epoch   0 Batch  400/781   test_loss = 0.884\n",
      "2018-09-14T15:13:03.114147: Epoch   0 Batch  450/781   test_loss = 0.845\n",
      "2018-09-14T15:13:04.411407: Epoch   0 Batch  500/781   test_loss = 0.809\n",
      "2018-09-14T15:13:05.746647: Epoch   0 Batch  550/781   test_loss = 0.869\n",
      "2018-09-14T15:13:07.034935: Epoch   0 Batch  600/781   test_loss = 0.881\n",
      "2018-09-14T15:13:08.367155: Epoch   0 Batch  650/781   test_loss = 0.794\n",
      "2018-09-14T15:13:09.619460: Epoch   0 Batch  700/781   test_loss = 0.914\n",
      "2018-09-14T15:13:11.272494: Epoch   0 Batch  750/781   test_loss = 0.848\n",
      "2018-09-14T15:13:14.288805: Epoch   1 Batch    0/3125   train_loss = 1.068\n",
      "2018-09-14T15:13:19.750659: Epoch   1 Batch   50/3125   train_loss = 0.897\n",
      "2018-09-14T15:13:25.023682: Epoch   1 Batch  100/3125   train_loss = 0.925\n",
      "2018-09-14T15:13:29.980852: Epoch   1 Batch  150/3125   train_loss = 0.915\n",
      "2018-09-14T15:13:34.540224: Epoch   1 Batch  200/3125   train_loss = 1.048\n",
      "2018-09-14T15:13:39.216584: Epoch   1 Batch  250/3125   train_loss = 0.840\n",
      "2018-09-14T15:13:43.556092: Epoch   1 Batch  300/3125   train_loss = 1.083\n",
      "2018-09-14T15:13:48.623192: Epoch   1 Batch  350/3125   train_loss = 0.827\n",
      "2018-09-14T15:13:52.789820: Epoch   1 Batch  400/3125   train_loss = 0.858\n",
      "2018-09-14T15:13:56.585680: Epoch   1 Batch  450/3125   train_loss = 0.823\n",
      "2018-09-14T15:14:00.381486: Epoch   1 Batch  500/3125   train_loss = 0.683\n",
      "2018-09-14T15:14:04.170350: Epoch   1 Batch  550/3125   train_loss = 0.904\n",
      "2018-09-14T15:14:08.078096: Epoch   1 Batch  600/3125   train_loss = 0.806\n",
      "2018-09-14T15:14:13.152202: Epoch   1 Batch  650/3125   train_loss = 0.862\n",
      "2018-09-14T15:14:17.839528: Epoch   1 Batch  700/3125   train_loss = 0.933\n",
      "2018-09-14T15:14:22.484886: Epoch   1 Batch  750/3125   train_loss = 0.844\n",
      "2018-09-14T15:14:27.211211: Epoch   1 Batch  800/3125   train_loss = 0.720\n",
      "2018-09-14T15:14:31.710625: Epoch   1 Batch  850/3125   train_loss = 0.965\n",
      "2018-09-14T15:14:37.199518: Epoch   1 Batch  900/3125   train_loss = 0.841\n",
      "2018-09-14T15:14:42.630390: Epoch   1 Batch  950/3125   train_loss = 0.840\n",
      "2018-09-14T15:14:47.930367: Epoch   1 Batch 1000/3125   train_loss = 0.946\n",
      "2018-09-14T15:14:53.349305: Epoch   1 Batch 1050/3125   train_loss = 0.831\n",
      "2018-09-14T15:14:57.190086: Epoch   1 Batch 1100/3125   train_loss = 0.806\n",
      "2018-09-14T15:15:00.632143: Epoch   1 Batch 1150/3125   train_loss = 0.838\n",
      "2018-09-14T15:15:03.778331: Epoch   1 Batch 1200/3125   train_loss = 0.919\n",
      "2018-09-14T15:15:06.373865: Epoch   1 Batch 1250/3125   train_loss = 0.882\n",
      "2018-09-14T15:15:08.775496: Epoch   1 Batch 1300/3125   train_loss = 0.839\n",
      "2018-09-14T15:15:11.170130: Epoch   1 Batch 1350/3125   train_loss = 0.804\n",
      "2018-09-14T15:15:13.586752: Epoch   1 Batch 1400/3125   train_loss = 0.914\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-09-14T15:15:17.968111: Epoch   1 Batch 1450/3125   train_loss = 0.881\n",
      "2018-09-14T15:15:23.030224: Epoch   1 Batch 1500/3125   train_loss = 0.883\n",
      "2018-09-14T15:15:28.084342: Epoch   1 Batch 1550/3125   train_loss = 0.822\n",
      "2018-09-14T15:15:33.129464: Epoch   1 Batch 1600/3125   train_loss = 0.764\n",
      "2018-09-14T15:15:38.173588: Epoch   1 Batch 1650/3125   train_loss = 0.741\n",
      "2018-09-14T15:15:43.179733: Epoch   1 Batch 1700/3125   train_loss = 0.813\n",
      "2018-09-14T15:15:48.221859: Epoch   1 Batch 1750/3125   train_loss = 0.763\n",
      "2018-09-14T15:15:53.283969: Epoch   1 Batch 1800/3125   train_loss = 0.778\n",
      "2018-09-14T15:15:58.775822: Epoch   1 Batch 1850/3125   train_loss = 0.837\n",
      "2018-09-14T15:16:04.748409: Epoch   1 Batch 1900/3125   train_loss = 0.728\n",
      "2018-09-14T15:16:10.657030: Epoch   1 Batch 1950/3125   train_loss = 0.777\n",
      "2018-09-14T15:16:16.677604: Epoch   1 Batch 2000/3125   train_loss = 1.016\n",
      "2018-09-14T15:16:22.674207: Epoch   1 Batch 2050/3125   train_loss = 0.910\n",
      "2018-09-14T15:16:28.407938: Epoch   1 Batch 2100/3125   train_loss = 0.789\n",
      "2018-09-14T15:16:34.315545: Epoch   1 Batch 2150/3125   train_loss = 0.898\n",
      "2018-09-14T15:16:40.333136: Epoch   1 Batch 2200/3125   train_loss = 0.805\n",
      "2018-09-14T15:16:46.239767: Epoch   1 Batch 2250/3125   train_loss = 0.930\n",
      "2018-09-14T15:16:52.368248: Epoch   1 Batch 2300/3125   train_loss = 0.848\n",
      "2018-09-14T15:16:58.480786: Epoch   1 Batch 2350/3125   train_loss = 0.920\n",
      "2018-09-14T15:17:04.549328: Epoch   1 Batch 2400/3125   train_loss = 0.856\n",
      "2018-09-14T15:17:10.570862: Epoch   1 Batch 2450/3125   train_loss = 0.893\n",
      "2018-09-14T15:17:16.896283: Epoch   1 Batch 2500/3125   train_loss = 0.791\n",
      "2018-09-14T15:17:22.805913: Epoch   1 Batch 2550/3125   train_loss = 0.915\n",
      "2018-09-14T15:17:28.944381: Epoch   1 Batch 2600/3125   train_loss = 0.823\n",
      "2018-09-14T15:17:34.960958: Epoch   1 Batch 2650/3125   train_loss = 0.742\n",
      "2018-09-14T15:17:41.083464: Epoch   1 Batch 2700/3125   train_loss = 0.867\n",
      "2018-09-14T15:17:47.270935: Epoch   1 Batch 2750/3125   train_loss = 0.963\n",
      "2018-09-14T15:17:52.952695: Epoch   1 Batch 2800/3125   train_loss = 1.025\n",
      "2018-09-14T15:17:58.940281: Epoch   1 Batch 2850/3125   train_loss = 0.905\n",
      "2018-09-14T15:18:05.028809: Epoch   1 Batch 2900/3125   train_loss = 0.803\n",
      "2018-09-14T15:18:11.046400: Epoch   1 Batch 2950/3125   train_loss = 0.968\n",
      "2018-09-14T15:18:16.874047: Epoch   1 Batch 3000/3125   train_loss = 0.925\n",
      "2018-09-14T15:18:22.721710: Epoch   1 Batch 3050/3125   train_loss = 0.797\n",
      "2018-09-14T15:18:28.759267: Epoch   1 Batch 3100/3125   train_loss = 1.002\n",
      "2018-09-14T15:18:31.692595: Epoch   1 Batch    0/781   test_loss = 0.832\n",
      "2018-09-14T15:18:33.771410: Epoch   1 Batch   50/781   test_loss = 0.840\n",
      "2018-09-14T15:18:35.755285: Epoch   1 Batch  100/781   test_loss = 1.015\n",
      "2018-09-14T15:18:37.734151: Epoch   1 Batch  150/781   test_loss = 0.888\n",
      "2018-09-14T15:18:39.572109: Epoch   1 Batch  200/781   test_loss = 0.885\n",
      "2018-09-14T15:18:41.405064: Epoch   1 Batch  250/781   test_loss = 0.807\n",
      "2018-09-14T15:18:43.226046: Epoch   1 Batch  300/781   test_loss = 0.804\n",
      "2018-09-14T15:18:45.054975: Epoch   1 Batch  350/781   test_loss = 0.866\n",
      "2018-09-14T15:18:46.894932: Epoch   1 Batch  400/781   test_loss = 0.857\n",
      "2018-09-14T15:18:48.757892: Epoch   1 Batch  450/781   test_loss = 0.794\n",
      "2018-09-14T15:18:50.607807: Epoch   1 Batch  500/781   test_loss = 0.772\n",
      "2018-09-14T15:18:52.451758: Epoch   1 Batch  550/781   test_loss = 0.837\n",
      "2018-09-14T15:18:54.256727: Epoch   1 Batch  600/781   test_loss = 0.848\n",
      "2018-09-14T15:18:56.070700: Epoch   1 Batch  650/781   test_loss = 0.760\n",
      "2018-09-14T15:18:57.885657: Epoch   1 Batch  700/781   test_loss = 0.883\n",
      "2018-09-14T15:18:59.856562: Epoch   1 Batch  750/781   test_loss = 0.800\n",
      "2018-09-14T15:19:02.956794: Epoch   2 Batch    0/3125   train_loss = 0.962\n",
      "2018-09-14T15:19:08.250751: Epoch   2 Batch   50/3125   train_loss = 0.821\n",
      "2018-09-14T15:19:13.457782: Epoch   2 Batch  100/3125   train_loss = 0.858\n",
      "2018-09-14T15:19:18.331023: Epoch   2 Batch  150/3125   train_loss = 0.855\n",
      "2018-09-14T15:19:23.099303: Epoch   2 Batch  200/3125   train_loss = 1.031\n",
      "2018-09-14T15:19:27.941542: Epoch   2 Batch  250/3125   train_loss = 0.803\n",
      "2018-09-14T15:19:32.872708: Epoch   2 Batch  300/3125   train_loss = 1.064\n",
      "2018-09-14T15:19:38.025790: Epoch   2 Batch  350/3125   train_loss = 0.800\n",
      "2018-09-14T15:19:42.910981: Epoch   2 Batch  400/3125   train_loss = 0.869\n",
      "2018-09-14T15:19:47.282482: Epoch   2 Batch  450/3125   train_loss = 0.795\n",
      "2018-09-14T15:19:51.761952: Epoch   2 Batch  500/3125   train_loss = 0.640\n",
      "2018-09-14T15:19:56.230404: Epoch   2 Batch  550/3125   train_loss = 0.864\n",
      "2018-09-14T15:20:00.675844: Epoch   2 Batch  600/3125   train_loss = 0.760\n",
      "2018-09-14T15:20:04.041927: Epoch   2 Batch  650/3125   train_loss = 0.839\n",
      "2018-09-14T15:20:06.950264: Epoch   2 Batch  700/3125   train_loss = 0.872\n",
      "2018-09-14T15:20:09.442845: Epoch   2 Batch  750/3125   train_loss = 0.791\n",
      "2018-09-14T15:20:11.912437: Epoch   2 Batch  800/3125   train_loss = 0.699\n",
      "2018-09-14T15:20:16.230004: Epoch   2 Batch  850/3125   train_loss = 0.935\n",
      "2018-09-14T15:20:22.009678: Epoch   2 Batch  900/3125   train_loss = 0.828\n",
      "2018-09-14T15:20:27.574533: Epoch   2 Batch  950/3125   train_loss = 0.827\n",
      "2018-09-14T15:20:33.300243: Epoch   2 Batch 1000/3125   train_loss = 0.897\n",
      "2018-09-14T15:20:37.017138: Epoch   2 Batch 1050/3125   train_loss = 0.787\n",
      "2018-09-14T15:20:39.976429: Epoch   2 Batch 1100/3125   train_loss = 0.739\n",
      "2018-09-14T15:20:43.395505: Epoch   2 Batch 1150/3125   train_loss = 0.797\n",
      "2018-09-14T15:20:48.358680: Epoch   2 Batch 1200/3125   train_loss = 0.911\n",
      "2018-09-14T15:20:53.377793: Epoch   2 Batch 1250/3125   train_loss = 0.841\n",
      "2018-09-14T15:20:58.411916: Epoch   2 Batch 1300/3125   train_loss = 0.768\n",
      "2018-09-14T15:21:03.981738: Epoch   2 Batch 1350/3125   train_loss = 0.776\n",
      "2018-09-14T15:21:09.693510: Epoch   2 Batch 1400/3125   train_loss = 0.826\n",
      "2018-09-14T15:21:14.759597: Epoch   2 Batch 1450/3125   train_loss = 0.852\n",
      "2018-09-14T15:21:18.401535: Epoch   2 Batch 1500/3125   train_loss = 0.871\n",
      "2018-09-14T15:21:21.438782: Epoch   2 Batch 1550/3125   train_loss = 0.789\n",
      "2018-09-14T15:21:24.348144: Epoch   2 Batch 1600/3125   train_loss = 0.750\n",
      "2018-09-14T15:21:27.211511: Epoch   2 Batch 1650/3125   train_loss = 0.743\n",
      "2018-09-14T15:21:31.809875: Epoch   2 Batch 1700/3125   train_loss = 0.811\n",
      "2018-09-14T15:21:36.844004: Epoch   2 Batch 1750/3125   train_loss = 0.741\n",
      "2018-09-14T15:21:41.958087: Epoch   2 Batch 1800/3125   train_loss = 0.747\n",
      "2018-09-14T15:21:47.313056: Epoch   2 Batch 1850/3125   train_loss = 0.758\n",
      "2018-09-14T15:21:52.814895: Epoch   2 Batch 1900/3125   train_loss = 0.693\n",
      "2018-09-14T15:21:57.796080: Epoch   2 Batch 1950/3125   train_loss = 0.797\n",
      "2018-09-14T15:22:03.013104: Epoch   2 Batch 2000/3125   train_loss = 0.939\n",
      "2018-09-14T15:22:08.863766: Epoch   2 Batch 2050/3125   train_loss = 0.853\n",
      "2018-09-14T15:22:14.796351: Epoch   2 Batch 2100/3125   train_loss = 0.792\n",
      "2018-09-14T15:22:20.125312: Epoch   2 Batch 2150/3125   train_loss = 0.857\n",
      "2018-09-14T15:22:25.708128: Epoch   2 Batch 2200/3125   train_loss = 0.815\n",
      "2018-09-14T15:22:31.621756: Epoch   2 Batch 2250/3125   train_loss = 0.916\n",
      "2018-09-14T15:22:37.704292: Epoch   2 Batch 2300/3125   train_loss = 0.833\n",
      "2018-09-14T15:22:42.899354: Epoch   2 Batch 2350/3125   train_loss = 0.915\n",
      "2018-09-14T15:22:48.086394: Epoch   2 Batch 2400/3125   train_loss = 0.826\n",
      "2018-09-14T15:22:53.213447: Epoch   2 Batch 2450/3125   train_loss = 0.869\n",
      "2018-09-14T15:22:58.367525: Epoch   2 Batch 2500/3125   train_loss = 0.756\n",
      "2018-09-14T15:23:02.764018: Epoch   2 Batch 2550/3125   train_loss = 0.894\n",
      "2018-09-14T15:23:06.441900: Epoch   2 Batch 2600/3125   train_loss = 0.778\n",
      "2018-09-14T15:23:09.771001: Epoch   2 Batch 2650/3125   train_loss = 0.715\n",
      "2018-09-14T15:23:13.151074: Epoch   2 Batch 2700/3125   train_loss = 0.859\n",
      "2018-09-14T15:23:18.480031: Epoch   2 Batch 2750/3125   train_loss = 0.988\n",
      "2018-09-14T15:23:24.468615: Epoch   2 Batch 2800/3125   train_loss = 0.998\n",
      "2018-09-14T15:23:30.650090: Epoch   2 Batch 2850/3125   train_loss = 0.850\n",
      "2018-09-14T15:23:36.602695: Epoch   2 Batch 2900/3125   train_loss = 0.770\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-09-14T15:23:42.370413: Epoch   2 Batch 2950/3125   train_loss = 0.948\n",
      "2018-09-14T15:23:47.766334: Epoch   2 Batch 3000/3125   train_loss = 0.922\n",
      "2018-09-14T15:23:52.650548: Epoch   2 Batch 3050/3125   train_loss = 0.764\n",
      "2018-09-14T15:23:56.999087: Epoch   2 Batch 3100/3125   train_loss = 0.981\n",
      "2018-09-14T15:23:58.775050: Epoch   2 Batch    0/781   test_loss = 0.841\n",
      "2018-09-14T15:24:00.415121: Epoch   2 Batch   50/781   test_loss = 0.820\n",
      "2018-09-14T15:24:02.215095: Epoch   2 Batch  100/781   test_loss = 0.998\n",
      "2018-09-14T15:24:03.974093: Epoch   2 Batch  150/781   test_loss = 0.864\n",
      "2018-09-14T15:24:05.728093: Epoch   2 Batch  200/781   test_loss = 0.855\n",
      "2018-09-14T15:24:07.648997: Epoch   2 Batch  250/781   test_loss = 0.804\n",
      "2018-09-14T15:24:09.419010: Epoch   2 Batch  300/781   test_loss = 0.782\n",
      "2018-09-14T15:24:11.165988: Epoch   2 Batch  350/781   test_loss = 0.833\n",
      "2018-09-14T15:24:12.940973: Epoch   2 Batch  400/781   test_loss = 0.873\n",
      "2018-09-14T15:24:14.690974: Epoch   2 Batch  450/781   test_loss = 0.811\n",
      "2018-09-14T15:24:16.553919: Epoch   2 Batch  500/781   test_loss = 0.777\n",
      "2018-09-14T15:24:18.278957: Epoch   2 Batch  550/781   test_loss = 0.805\n",
      "2018-09-14T15:24:20.043921: Epoch   2 Batch  600/781   test_loss = 0.847\n",
      "2018-09-14T15:24:21.837898: Epoch   2 Batch  650/781   test_loss = 0.747\n",
      "2018-09-14T15:24:23.548922: Epoch   2 Batch  700/781   test_loss = 0.872\n",
      "2018-09-14T15:24:25.317914: Epoch   2 Batch  750/781   test_loss = 0.812\n",
      "2018-09-14T15:24:28.180310: Epoch   3 Batch    0/3125   train_loss = 0.902\n",
      "2018-09-14T15:24:33.554214: Epoch   3 Batch   50/3125   train_loss = 0.821\n",
      "2018-09-14T15:24:38.873211: Epoch   3 Batch  100/3125   train_loss = 0.878\n",
      "2018-09-14T15:24:44.327101: Epoch   3 Batch  150/3125   train_loss = 0.826\n",
      "2018-09-14T15:24:49.591068: Epoch   3 Batch  200/3125   train_loss = 0.978\n",
      "2018-09-14T15:24:54.183477: Epoch   3 Batch  250/3125   train_loss = 0.766\n",
      "2018-09-14T15:24:57.896334: Epoch   3 Batch  300/3125   train_loss = 1.000\n",
      "2018-09-14T15:25:00.920626: Epoch   3 Batch  350/3125   train_loss = 0.770\n",
      "2018-09-14T15:25:04.895368: Epoch   3 Batch  400/3125   train_loss = 0.870\n",
      "2018-09-14T15:25:09.442774: Epoch   3 Batch  450/3125   train_loss = 0.782\n",
      "2018-09-14T15:25:14.186047: Epoch   3 Batch  500/3125   train_loss = 0.637\n",
      "2018-09-14T15:25:19.421062: Epoch   3 Batch  550/3125   train_loss = 0.819\n",
      "2018-09-14T15:25:24.265293: Epoch   3 Batch  600/3125   train_loss = 0.745\n",
      "2018-09-14T15:25:29.933090: Epoch   3 Batch  650/3125   train_loss = 0.838\n",
      "2018-09-14T15:25:34.349566: Epoch   3 Batch  700/3125   train_loss = 0.853\n",
      "2018-09-14T15:25:38.179356: Epoch   3 Batch  750/3125   train_loss = 0.762\n",
      "2018-09-14T15:25:41.976197: Epoch   3 Batch  800/3125   train_loss = 0.699\n",
      "2018-09-14T15:25:45.772051: Epoch   3 Batch  850/3125   train_loss = 0.876\n",
      "2018-09-14T15:25:49.449949: Epoch   3 Batch  900/3125   train_loss = 0.799\n",
      "2018-09-14T15:25:52.422237: Epoch   3 Batch  950/3125   train_loss = 0.788\n",
      "2018-09-14T15:25:55.372550: Epoch   3 Batch 1000/3125   train_loss = 0.878\n",
      "2018-09-14T15:25:58.384831: Epoch   3 Batch 1050/3125   train_loss = 0.762\n",
      "2018-09-14T15:26:00.949373: Epoch   3 Batch 1100/3125   train_loss = 0.736\n",
      "2018-09-14T15:26:03.447948: Epoch   3 Batch 1150/3125   train_loss = 0.783\n",
      "2018-09-14T15:26:06.088456: Epoch   3 Batch 1200/3125   train_loss = 0.873\n",
      "2018-09-14T15:26:08.506076: Epoch   3 Batch 1250/3125   train_loss = 0.838\n",
      "2018-09-14T15:26:10.878708: Epoch   3 Batch 1300/3125   train_loss = 0.769\n",
      "2018-09-14T15:26:13.162406: Epoch   3 Batch 1350/3125   train_loss = 0.752\n",
      "2018-09-14T15:26:15.331168: Epoch   3 Batch 1400/3125   train_loss = 0.805\n",
      "2018-09-14T15:26:17.503931: Epoch   3 Batch 1450/3125   train_loss = 0.815\n",
      "2018-09-14T15:26:21.823473: Epoch   3 Batch 1500/3125   train_loss = 0.856\n",
      "2018-09-14T15:26:26.848604: Epoch   3 Batch 1550/3125   train_loss = 0.765\n",
      "2018-09-14T15:26:31.808801: Epoch   3 Batch 1600/3125   train_loss = 0.742\n",
      "2018-09-14T15:26:36.846904: Epoch   3 Batch 1650/3125   train_loss = 0.749\n",
      "2018-09-14T15:26:41.781090: Epoch   3 Batch 1700/3125   train_loss = 0.786\n",
      "2018-09-14T15:26:46.722272: Epoch   3 Batch 1750/3125   train_loss = 0.724\n",
      "2018-09-14T15:26:51.631465: Epoch   3 Batch 1800/3125   train_loss = 0.729\n",
      "2018-09-14T15:26:56.581648: Epoch   3 Batch 1850/3125   train_loss = 0.729\n",
      "2018-09-14T15:27:01.582791: Epoch   3 Batch 1900/3125   train_loss = 0.663\n",
      "2018-09-14T15:27:06.794848: Epoch   3 Batch 1950/3125   train_loss = 0.766\n",
      "2018-09-14T15:27:12.088827: Epoch   3 Batch 2000/3125   train_loss = 0.922\n",
      "2018-09-14T15:27:17.352797: Epoch   3 Batch 2050/3125   train_loss = 0.829\n",
      "2018-09-14T15:27:22.603836: Epoch   3 Batch 2100/3125   train_loss = 0.769\n",
      "2018-09-14T15:27:27.734905: Epoch   3 Batch 2150/3125   train_loss = 0.818\n",
      "2018-09-14T15:27:33.019892: Epoch   3 Batch 2200/3125   train_loss = 0.813\n",
      "2018-09-14T15:27:38.278891: Epoch   3 Batch 2250/3125   train_loss = 0.923\n",
      "2018-09-14T15:27:43.489919: Epoch   3 Batch 2300/3125   train_loss = 0.784\n",
      "2018-09-14T15:27:48.676937: Epoch   3 Batch 2350/3125   train_loss = 0.895\n",
      "2018-09-14T15:27:53.808034: Epoch   3 Batch 2400/3125   train_loss = 0.810\n",
      "2018-09-14T15:27:59.047017: Epoch   3 Batch 2450/3125   train_loss = 0.857\n",
      "2018-09-14T15:28:04.202076: Epoch   3 Batch 2500/3125   train_loss = 0.733\n",
      "2018-09-14T15:28:07.957958: Epoch   3 Batch 2550/3125   train_loss = 0.882\n",
      "2018-09-14T15:28:11.746797: Epoch   3 Batch 2600/3125   train_loss = 0.778\n",
      "2018-09-14T15:28:15.580586: Epoch   3 Batch 2650/3125   train_loss = 0.706\n",
      "2018-09-14T15:28:19.393435: Epoch   3 Batch 2700/3125   train_loss = 0.886\n",
      "2018-09-14T15:28:23.235244: Epoch   3 Batch 2750/3125   train_loss = 0.931\n",
      "2018-09-14T15:28:27.021067: Epoch   3 Batch 2800/3125   train_loss = 0.974\n",
      "2018-09-14T15:28:30.125290: Epoch   3 Batch 2850/3125   train_loss = 0.811\n",
      "2018-09-14T15:28:33.192540: Epoch   3 Batch 2900/3125   train_loss = 0.755\n",
      "2018-09-14T15:28:36.269807: Epoch   3 Batch 2950/3125   train_loss = 0.927\n",
      "2018-09-14T15:28:39.238092: Epoch   3 Batch 3000/3125   train_loss = 0.901\n",
      "2018-09-14T15:28:42.279362: Epoch   3 Batch 3050/3125   train_loss = 0.735\n",
      "2018-09-14T15:28:45.240670: Epoch   3 Batch 3100/3125   train_loss = 0.969\n",
      "2018-09-14T15:28:46.711851: Epoch   3 Batch    0/781   test_loss = 0.817\n",
      "2018-09-14T15:28:48.660726: Epoch   3 Batch   50/781   test_loss = 0.802\n",
      "2018-09-14T15:28:50.439706: Epoch   3 Batch  100/781   test_loss = 0.975\n",
      "2018-09-14T15:28:52.208697: Epoch   3 Batch  150/781   test_loss = 0.838\n",
      "2018-09-14T15:28:53.924749: Epoch   3 Batch  200/781   test_loss = 0.823\n",
      "2018-09-14T15:28:55.694708: Epoch   3 Batch  250/781   test_loss = 0.798\n",
      "2018-09-14T15:28:57.549653: Epoch   3 Batch  300/781   test_loss = 0.777\n",
      "2018-09-14T15:28:59.284662: Epoch   3 Batch  350/781   test_loss = 0.837\n",
      "2018-09-14T15:29:00.997685: Epoch   3 Batch  400/781   test_loss = 0.866\n",
      "2018-09-14T15:29:02.825642: Epoch   3 Batch  450/781   test_loss = 0.796\n",
      "2018-09-14T15:29:04.559682: Epoch   3 Batch  500/781   test_loss = 0.734\n",
      "2018-09-14T15:29:06.265681: Epoch   3 Batch  550/781   test_loss = 0.775\n",
      "2018-09-14T15:29:08.067664: Epoch   3 Batch  600/781   test_loss = 0.852\n",
      "2018-09-14T15:29:09.866637: Epoch   3 Batch  650/781   test_loss = 0.726\n",
      "2018-09-14T15:29:11.713575: Epoch   3 Batch  700/781   test_loss = 0.852\n",
      "2018-09-14T15:29:13.557529: Epoch   3 Batch  750/781   test_loss = 0.805\n",
      "2018-09-14T15:29:16.325972: Epoch   4 Batch    0/3125   train_loss = 0.888\n",
      "2018-09-14T15:29:21.754875: Epoch   4 Batch   50/3125   train_loss = 0.812\n",
      "2018-09-14T15:29:26.908914: Epoch   4 Batch  100/3125   train_loss = 0.872\n",
      "2018-09-14T15:29:32.155944: Epoch   4 Batch  150/3125   train_loss = 0.821\n",
      "2018-09-14T15:29:37.389928: Epoch   4 Batch  200/3125   train_loss = 0.950\n",
      "2018-09-14T15:29:42.621945: Epoch   4 Batch  250/3125   train_loss = 0.747\n",
      "2018-09-14T15:29:47.799022: Epoch   4 Batch  300/3125   train_loss = 0.985\n",
      "2018-09-14T15:29:53.039034: Epoch   4 Batch  350/3125   train_loss = 0.767\n",
      "2018-09-14T15:29:58.453916: Epoch   4 Batch  400/3125   train_loss = 0.857\n",
      "2018-09-14T15:30:03.756921: Epoch   4 Batch  450/3125   train_loss = 0.772\n",
      "2018-09-14T15:30:09.035886: Epoch   4 Batch  500/3125   train_loss = 0.620\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2018-09-14T15:30:14.271929: Epoch   4 Batch  550/3125   train_loss = 0.791\n",
      "2018-09-14T15:30:19.715795: Epoch   4 Batch  600/3125   train_loss = 0.748\n",
      "2018-09-14T15:30:24.334161: Epoch   4 Batch  650/3125   train_loss = 0.819\n",
      "2018-09-14T15:30:29.016491: Epoch   4 Batch  700/3125   train_loss = 0.821\n",
      "2018-09-14T15:30:33.808751: Epoch   4 Batch  750/3125   train_loss = 0.755\n",
      "2018-09-14T15:30:38.886883: Epoch   4 Batch  800/3125   train_loss = 0.681\n",
      "2018-09-14T15:30:43.818047: Epoch   4 Batch  850/3125   train_loss = 0.862\n",
      "2018-09-14T15:30:48.872166: Epoch   4 Batch  900/3125   train_loss = 0.765\n",
      "2018-09-14T15:30:54.565912: Epoch   4 Batch  950/3125   train_loss = 0.777\n",
      "2018-09-14T15:31:00.504526: Epoch   4 Batch 1000/3125   train_loss = 0.853\n",
      "2018-09-14T15:31:06.372210: Epoch   4 Batch 1050/3125   train_loss = 0.739\n",
      "2018-09-14T15:31:12.276819: Epoch   4 Batch 1100/3125   train_loss = 0.732\n",
      "2018-09-14T15:31:18.005545: Epoch   4 Batch 1150/3125   train_loss = 0.762\n",
      "2018-09-14T15:31:22.870797: Epoch   4 Batch 1200/3125   train_loss = 0.842\n",
      "2018-09-14T15:31:27.452180: Epoch   4 Batch 1250/3125   train_loss = 0.826\n",
      "2018-09-14T15:31:31.120085: Epoch   4 Batch 1300/3125   train_loss = 0.756\n",
      "2018-09-14T15:31:34.396199: Epoch   4 Batch 1350/3125   train_loss = 0.735\n",
      "2018-09-14T15:31:37.130654: Epoch   4 Batch 1400/3125   train_loss = 0.788\n",
      "2018-09-14T15:31:39.594234: Epoch   4 Batch 1450/3125   train_loss = 0.797\n",
      "2018-09-14T15:31:41.957902: Epoch   4 Batch 1500/3125   train_loss = 0.841\n",
      "2018-09-14T15:31:45.687787: Epoch   4 Batch 1550/3125   train_loss = 0.739\n",
      "2018-09-14T15:31:50.679916: Epoch   4 Batch 1600/3125   train_loss = 0.727\n",
      "2018-09-14T15:31:55.699078: Epoch   4 Batch 1650/3125   train_loss = 0.728\n",
      "2018-09-14T15:32:00.895082: Epoch   4 Batch 1700/3125   train_loss = 0.771\n",
      "2018-09-14T15:32:06.159079: Epoch   4 Batch 1750/3125   train_loss = 0.722\n",
      "2018-09-14T15:32:10.362704: Epoch   4 Batch 1800/3125   train_loss = 0.709\n",
      "2018-09-14T15:32:13.596859: Epoch   4 Batch 1850/3125   train_loss = 0.730\n",
      "2018-09-14T15:32:16.689078: Epoch   4 Batch 1900/3125   train_loss = 0.634\n",
      "2018-09-14T15:32:19.281600: Epoch   4 Batch 1950/3125   train_loss = 0.765\n",
      "2018-09-14T15:32:21.647250: Epoch   4 Batch 2000/3125   train_loss = 0.930\n",
      "2018-09-14T15:32:24.036888: Epoch   4 Batch 2050/3125   train_loss = 0.826\n",
      "2018-09-14T15:32:26.519680: Epoch   4 Batch 2100/3125   train_loss = 0.770\n",
      "2018-09-14T15:32:29.047238: Epoch   4 Batch 2150/3125   train_loss = 0.795\n",
      "2018-09-14T15:32:31.647754: Epoch   4 Batch 2200/3125   train_loss = 0.791\n",
      "2018-09-14T15:32:34.361207: Epoch   4 Batch 2250/3125   train_loss = 0.920\n",
      "2018-09-14T15:32:37.501429: Epoch   4 Batch 2300/3125   train_loss = 0.772\n",
      "2018-09-14T15:32:43.457026: Epoch   4 Batch 2350/3125   train_loss = 0.895\n",
      "2018-09-14T15:32:47.524682: Epoch   4 Batch 2400/3125   train_loss = 0.787\n",
      "2018-09-14T15:32:52.618807: Epoch   4 Batch 2450/3125   train_loss = 0.847\n",
      "2018-09-14T15:32:57.647939: Epoch   4 Batch 2500/3125   train_loss = 0.724\n",
      "2018-09-14T15:33:02.440195: Epoch   4 Batch 2550/3125   train_loss = 0.878\n",
      "2018-09-14T15:33:05.938200: Epoch   4 Batch 2600/3125   train_loss = 0.768\n",
      "2018-09-14T15:33:09.412219: Epoch   4 Batch 2650/3125   train_loss = 0.700\n",
      "2018-09-14T15:33:12.676357: Epoch   4 Batch 2700/3125   train_loss = 0.883\n",
      "2018-09-14T15:33:16.096407: Epoch   4 Batch 2750/3125   train_loss = 0.913\n",
      "2018-09-14T15:33:21.820152: Epoch   4 Batch 2800/3125   train_loss = 0.959\n",
      "2018-09-14T15:33:27.594859: Epoch   4 Batch 2850/3125   train_loss = 0.816\n",
      "2018-09-14T15:33:33.413541: Epoch   4 Batch 2900/3125   train_loss = 0.713\n",
      "2018-09-14T15:33:39.301154: Epoch   4 Batch 2950/3125   train_loss = 0.890\n",
      "2018-09-14T15:33:44.963953: Epoch   4 Batch 3000/3125   train_loss = 0.884\n",
      "2018-09-14T15:33:50.223929: Epoch   4 Batch 3050/3125   train_loss = 0.708\n",
      "2018-09-14T15:33:54.874277: Epoch   4 Batch 3100/3125   train_loss = 0.946\n",
      "2018-09-14T15:33:57.452819: Epoch   4 Batch    0/781   test_loss = 0.825\n",
      "2018-09-14T15:33:59.345721: Epoch   4 Batch   50/781   test_loss = 0.779\n",
      "2018-09-14T15:34:01.333590: Epoch   4 Batch  100/781   test_loss = 0.947\n",
      "2018-09-14T15:34:03.147583: Epoch   4 Batch  150/781   test_loss = 0.811\n",
      "2018-09-14T15:34:04.919550: Epoch   4 Batch  200/781   test_loss = 0.797\n",
      "2018-09-14T15:34:06.714525: Epoch   4 Batch  250/781   test_loss = 0.784\n",
      "2018-09-14T15:34:08.498509: Epoch   4 Batch  300/781   test_loss = 0.755\n",
      "2018-09-14T15:34:10.298476: Epoch   4 Batch  350/781   test_loss = 0.822\n",
      "2018-09-14T15:34:12.156420: Epoch   4 Batch  400/781   test_loss = 0.827\n",
      "2018-09-14T15:34:13.933431: Epoch   4 Batch  450/781   test_loss = 0.786\n",
      "2018-09-14T15:34:15.785347: Epoch   4 Batch  500/781   test_loss = 0.718\n",
      "2018-09-14T15:34:17.568328: Epoch   4 Batch  550/781   test_loss = 0.764\n",
      "2018-09-14T15:34:19.427269: Epoch   4 Batch  600/781   test_loss = 0.849\n",
      "2018-09-14T15:34:21.344205: Epoch   4 Batch  650/781   test_loss = 0.697\n",
      "2018-09-14T15:34:23.180135: Epoch   4 Batch  700/781   test_loss = 0.842\n",
      "2018-09-14T15:34:24.983118: Epoch   4 Batch  750/781   test_loss = 0.792\n",
      "Model Trained and Saved\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "import matplotlib.pyplot as plt\n",
    "import time\n",
    "import datetime\n",
    "\n",
    "losses = {'train':[], 'test':[]}\n",
    "\n",
    "with tf.Session(graph=train_graph) as sess:\n",
    "    \n",
    "    #搜集数据给tensorBoard用\n",
    "    # Keep track of gradient values and sparsity\n",
    "    grad_summaries = []\n",
    "    for g, v in gradients:\n",
    "        if g is not None:\n",
    "            grad_hist_summary = tf.summary.histogram(\"{}/grad/hist\".format(v.name.replace(':', '_')), g)\n",
    "            # tf.nn.zero_fraction 用于计算矩阵中 0 所占的比重，也就是计算矩阵的稀疏程度\n",
    "            sparsity_summary = tf.summary.scalar(\"{}/grad/sparsity\".format(v.name.replace(':', '_')), tf.nn.zero_fraction(g))\n",
    "            grad_summaries.append(grad_hist_summary)\n",
    "            grad_summaries.append(sparsity_summary)\n",
    "    grad_summaries_merged = tf.summary.merge(grad_summaries)\n",
    "        \n",
    "    # Output directory for models and summaries\n",
    "    timestamp = str(int(time.time()))\n",
    "    out_dir = os.path.abspath(os.path.join(os.path.curdir, \"runs\", timestamp))\n",
    "    print(\"Writing to {}\\n\".format(out_dir))\n",
    "     \n",
    "    # Summaries for loss and accuracy\n",
    "    loss_summary = tf.summary.scalar(\"loss\", loss)\n",
    "\n",
    "    # Train Summaries\n",
    "    train_summary_op = tf.summary.merge([loss_summary, grad_summaries_merged])\n",
    "    train_summary_dir = os.path.join(out_dir, \"summaries\", \"train\")\n",
    "    train_summary_writer = tf.summary.FileWriter(train_summary_dir, sess.graph)\n",
    "\n",
    "    # Inference summaries\n",
    "    inference_summary_op = tf.summary.merge([loss_summary])\n",
    "    inference_summary_dir = os.path.join(out_dir, \"summaries\", \"inference\")\n",
    "    inference_summary_writer = tf.summary.FileWriter(inference_summary_dir, sess.graph)\n",
    "\n",
    "    sess.run(tf.global_variables_initializer())\n",
    "    saver = tf.train.Saver()\n",
    "    for epoch_i in range(num_epochs):\n",
    "        \n",
    "        #将数据集分成训练集和测试集，随机种子不固定\n",
    "        train_X,test_X, train_y, test_y = train_test_split(features,  \n",
    "                                                           targets_values,  \n",
    "                                                           test_size = 0.2,  \n",
    "                                                           random_state = 0)  \n",
    "        \n",
    "        train_batches = get_batches(train_X, train_y, batch_size)\n",
    "        test_batches = get_batches(test_X, test_y, batch_size)\n",
    "    \n",
    "        #训练的迭代，保存训练损失\n",
    "        for batch_i in range(len(train_X) // batch_size):\n",
    "            x, y = next(train_batches)\n",
    "\n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: dropout_keep, #dropout_keep\n",
    "                lr: learning_rate}\n",
    "\n",
    "            step, train_loss, summaries, _ = sess.run([global_step, loss, train_summary_op, train_op], feed)  #cost\n",
    "            losses['train'].append(train_loss)\n",
    "            train_summary_writer.add_summary(summaries, step)  #\n",
    "            \n",
    "            # Show every <show_every_n_batches> batches\n",
    "            if batch_i % show_every_n_batches == 0:\n",
    "                time_str = datetime.datetime.now().isoformat()\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   train_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(train_X) // batch_size),\n",
    "                    train_loss))\n",
    "                \n",
    "        #使用测试数据的迭代\n",
    "        for batch_i  in range(len(test_X) // batch_size):\n",
    "            x, y = next(test_batches)\n",
    "            \n",
    "            categories = np.zeros([batch_size, 18])\n",
    "            for i in range(batch_size):\n",
    "                categories[i] = x.take(6,1)[i]\n",
    "\n",
    "            titles = np.zeros([batch_size, sentences_size])\n",
    "            for i in range(batch_size):\n",
    "                titles[i] = x.take(5,1)[i]\n",
    "\n",
    "            feed = {\n",
    "                uid: np.reshape(x.take(0,1), [batch_size, 1]),\n",
    "                user_gender: np.reshape(x.take(2,1), [batch_size, 1]),\n",
    "                user_age: np.reshape(x.take(3,1), [batch_size, 1]),\n",
    "                user_job: np.reshape(x.take(4,1), [batch_size, 1]),\n",
    "                movie_id: np.reshape(x.take(1,1), [batch_size, 1]),\n",
    "                movie_categories: categories,  #x.take(6,1)\n",
    "                movie_titles: titles,  #x.take(5,1)\n",
    "                targets: np.reshape(y, [batch_size, 1]),\n",
    "                dropout_keep_prob: 1,\n",
    "                lr: learning_rate}\n",
    "            \n",
    "            step, test_loss, summaries = sess.run([global_step, loss, inference_summary_op], feed)  #cost\n",
    "\n",
    "            #保存测试损失\n",
    "            losses['test'].append(test_loss)\n",
    "            inference_summary_writer.add_summary(summaries, step)  #\n",
    "\n",
    "            time_str = datetime.datetime.now().isoformat()\n",
    "            if batch_i % show_every_n_batches == 0:\n",
    "                print('{}: Epoch {:>3} Batch {:>4}/{}   test_loss = {:.3f}'.format(\n",
    "                    time_str,\n",
    "                    epoch_i,\n",
    "                    batch_i,\n",
    "                    (len(test_X) // batch_size),\n",
    "                    test_loss))\n",
    "\n",
    "    # Save Model\n",
    "    saver.save(sess, save_dir)  #, global_step=epoch_i\n",
    "    print('Model Trained and Saved')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 在 TensorBoard 中查看可视化结果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "tensorboard --logdir /PATH_TO_CODE/runs/1513402825/summaries/\n",
    "\n",
    "<img src=\"assets/loss.png\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 保存参数\n",
    "保存`save_dir` 在生成预测时使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "save_params((save_dir))\n",
    "\n",
    "load_dir = load_params()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示训练Loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAH0CAYAAABfKsnMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd8VfX9x/H3N5MkhEAIe+89JAwZKriooIgDq1VrraOtWiduqNQ6UGtb18/VirRqVRSRgoqKyJAdUGTIDhB2CISQndzv748k14QkEODce3LC6/l45HHvPefccz6J3PZ9v+c7jLVWAAAAALwlxO0CAAAAAJw4gjwAAADgQQR5AAAAwIMI8gAAAIAHEeQBAAAADyLIAwAAAB5EkAcAAAA8iCAPAAAAeBBBHgAAAPAggjwAAADgQQR5AAAAwIMI8gAAAIAHEeQBAAAADyLIAwAAAB5EkAcAAAA8iCAPAAAAeFCY2wUEgjFmq6Q6kpJdLgUAAAA1W2tJh621bYJ94RoZ5CXViYqKiu/SpUu824UAAACg5lq3bp2ys7NduXZNDfLJXbp0iU9KSnK7DgAAANRgiYmJWrFiRbIb16aPPAAAAOBBBHkAAADAgwjyAAAAgAcR5AEAAAAPIsgDAAAAHkSQBwAAADyIIA8AAAB4UE2dRx4AAFSBz+dTWlqaMjIylJubK2ut2yUBrjHGKDIyUrGxsYqPj1dISPVu8ybIAwBwmvL5fNqxY4eysrLcLgWoFqy1ysnJUU5OjjIzM9WiRYtqHeYJ8gAAnKbS0tKUlZWlsLAwNW7cWDExMdU6tACB5vP5lJmZqT179igrK0tpaWlKSEhwu6xK8WkFAOA0lZGRIUlq3LixYmNjCfE47YWEhCg2NlaNGzeW9PNnpLriEwsAwGkqNzdXkhQTE+NyJUD1UvKZKPmMVFcEeQAATlMlA1tpiQfKMsZIUrUf/M0nFwAAACilJMhXdwR5AAAAwIOYtcYh1loV+n6+/RIWynckAAAABA5p0yE5+T61f/RztX/0c3WfMMvtcgAAgIccOXJExhhdfPHFp3yuvn37qnbt2g5U5ZyXX35Zxhh99NFHbpdSoxDkAQDAacsYc0I/b7/9ttslA350rQEAAKetxx57rNy2f/zjH0pPT9ddd92lunXrltnXu3fvgNQRExOjdevWOdKS/vHHH1f7aRPhDII8AAA4bU2YMKHctrffflvp6em6++671bp166DUYYxR586dHTlXq1atHDkPqj+61gAAAJygkn7o2dnZGjdunNq3b6+IiAjdcccdkqQDBw5o4sSJOuecc9S0aVNFRESoUaNGuuKKK7RixYpy56usj/zYsWNljNHy5cv17rvvKjExUVFRUUpISND111+vffv2VVpbaTNmzJAxRn/961+1dOlSDR8+XHFxcapdu7bOP/98JSUlVfh7bt++Xdddd50SEhIUHR2txMREffDBB2XOd6oWLVqkSy+9VAkJCYqMjFTbtm119913a//+/eWO3bVrl+666y517NhR0dHRqlevnrp06aKbbrpJO3bs8B/n8/n05ptvasCAAUpISFBUVJRatmypESNGaNq0aadcc3VBi3wAVPO1AwAAgAN8Pp8uvvhirV+/XsOHD1f9+vX9reErV67UY489pqFDh+rSSy9VXFyctm7dqunTp2vGjBn66quvdPbZZ1f5Ws8++6xmzJihSy+9VMOGDdN3332nd955R6tXr9by5csVGhpapfMsWLBA48aN09ChQ3XLLbdoy5YtmjZtmoYOHarVq1eXac1PSUnRwIEDtWvXLp133nnq16+fdu7cqRtuuEEXXXTRif2xKvHhhx/q2muvVWhoqMaMGaPmzZtr8eLFeuGFF/Tpp5/qu+++U9OmTSVJhw8f1oABA7Rr1y5deOGFGj16tPLz87Vt2zZ99NFHuv7669WiRQtJ0t13362XXnpJHTp00DXXXKPatWtr165dWrJkiaZNm6bRo0c7Ur/bCPIO8ci6AQAAwCHZ2dnKyMjQ6tWry/Wl79Onj/bs2aN69eqV2b5582YNGDBA9913n5YtW1bla82ePVvff/+9OnbsKKlo2uvRo0dr+vTpmjVrlkaMGFGl83z66aeaMmWKrrzySv+2559/XmPHjtUrr7yiZ5991r/9vvvu065du/T4449r/Pjx/u233XabhgwZUuXaK5OWlqabb75ZxhgtWLBAffv29e8bP368nnjiCd1xxx2aOnWqJGnmzJlKSUnRuHHj9Je//KXMuXJyclRQUCDp59b4du3a6ccff1RkZGSZY1NTU0+59uqCIA8AACrU+qGZbpdQZckTR7py3aeffrpciJek+Pj4Co9v166dRo0apUmTJunAgQOqX79+la5z//33+0O8VNSn/uabb9b06dO1dOnSKgf54cOHlwnxknTrrbdq7NixWrp0qX9bRkaGpk6dqoYNG+r+++8vc/yZZ56pMWPG6P3336/SNSszZcoUZWRk6JZbbikT4iXp0Ucf1T//+U99+umnSk1NVUJCgn9fVFRUuXPVqlWrzGtjjCIiIiq8U1H6XF5HH3kAAICT1L9//0r3zZkzR5dffrmaN2+uiIgI/xSWkyZNklTU37uqjg66kvzdSA4ePHhK54mNjVVcXFyZ86xevVoFBQVKTEwsF5IlOdIiXzJW4Nxzzy23r1atWho0aJB8Pp9++OEHSdIFF1ygBg0aaPz48br44ov1yiuv6Pvvv5fP5yvz3pCQEF199dVat26dunfvrvHjx+vLL79URkbGKddc3dAiDwAAcBKio6MVGxtb4b533nlHv/71r1W7dm1dcMEFatOmjWJiYmSM0ZdffqlFixad0BSRFbX6h4UVxbjCwsJTOk/JuUqfJz09XZLUqFGjCo+vbPuJKLlGkyZNKtxfsv3QoUOSilrSlyxZogkTJmjGjBmaOXOmv5Y777xTDz74oL8F/vXXX1fnzp01efJkPfHEE5Kk8PBwjRo1Ss8//3yNmdmHIB8AjHUFANQEbnVX8QpzjAFy48aNU2xsrFauXKm2bduW2bdx40YtWrQo0OWdkjp16kiS9u7dW+H+yrafiLi4OEnSnj17Kty/e/fuMsdJUps2bTR58mT5fD6tXr1as2fP1ssvv6xHH31UoaGhevDBByUVhfYHHnhADzzwgPbs2aP58+frnXfe0ccff6yffvpJP/zwQ5UHCFdndK0BAABwUEFBgbZt26bevXuXC/H5+fnVPsRLUo8ePRQWFqakpCTl5OSU279gwYJTvsYZZ5whSfr222/L7cvNzdWiRYtkjKlwEa6QkBD17NlT99xzj2bMmCFJlU4r2bhxY40ZM0affvqp+vfvrzVr1mjTpk2nXH91QJAHAABwUFhYmJo1a6Y1a9aUmSHF5/Pp4Ycf1tatW12srmpiY2M1evRo7du3T88991yZfUuWLNGUKVNO+RpXXXWVateurUmTJvn7wZd4+umntXv3bv/88pL0/fffKyUlpdx5Su4OREdHSyqak3/u3LnljsvNzfV356lowKwX0bUGAADAYffcc4/Gjh2rnj176vLLL1dISIjmzp2r5ORkXXTRRfr888/dLvG4nn/+eS1YsEB/+tOfNG/ePPXr108pKSn68MMPdckll2jatGkKCTn5NuH4+Hi98cYbuv766zVw4ECNGTNGzZo10+LFizVnzhy1aNFCL7/8sv/4GTNm6LHHHtOQIUPUqVMnJSQkaNu2bfr0008VGhqqsWPHSirqUz906FC1a9dO/fv3V8uWLZWVlaUvvvhCGzdu1K9+9Su1bNnylP8+1QFBHgAAwGH33nuvateurZdffllvvfWWYmJiNHToUH344Yd68803PRHkW7ZsqcWLF+vhhx/WrFmztGDBAnXt2lWTJ09Wdna2pk2b5u9Lf7KuueYatWzZUhMnTtSMGTOUkZGhpk2b6o9//KPGjRunhg0b+o8dNWqU9u/fr/nz52vq1Kk6cuSImjRpoksuuUT33Xeff0ae+vXr66mnntKcOXM0f/587d+/X3Xq1FGHDh304IMP6oYbbjilmqsTYx1YhtQYc6WkcyT1ltRLUqykd62111Xx/f+S9Nvilx2stafUcckYk9SnT58+lS03HAg5+YXqPP4LSVJEaIg2POnMimcAAATKunXrJEldunRxuRJ4zV133aUXX3xRCxYs0ODBg90uJyCq+vlITEzUihUrVlhrE4NRV2lO9ZEfJ+kOFQX5nSfyRmPMJSoK8UccqsUVrOwKAABqmormul+2bJneeOMNNW3aVAMGDHChKpRwqmvNPZJSJG1SUcv8nKq8yRjTQNKbkj6Q1Lj4vQAAAKgGunTpoj59+qhbt26qVauW1q9f7+8W9Morr/jnsoc7HPnrW2v9wf1Yc6pW4I3ix9slfexELQAAAHDGbbfdps8++0zvvvuujhw5onr16uniiy/WAw88oEGDBrld3mnPta9RxpjfSBot6TJr7YET/AIAAACAAHv66af19NNPu10GKuFKkDfGtJL0gqR3rLUVz95ftfNUNpq188me0wmWtV0BAAAQYEFfEMoYEyJpsooGt94Z7OsHihF3FAAAABA8brTI36OiQa0jrbUHT+VElU3zU9xS3+dUzg0AAIDTkxPTswdDUFvkjTEdJD0paZK19rNgXhsAAJRVMj7N5/O5XAlQvZQE+eo+hjPYXWu6SYqUdKMxxpb+0c9TT24s3jY6yLUBAHBaiYyMlCRlZma6XAlQvZR8Jko+I9VVsLvWJEv6VyX7RqpoLvkpkg4XHwsAAAIkNjZWOTk52rNnjyQpJiZGxphq3woJBIK1VtZaZWZm+j8TsbGxLld1bEEN8tba7yXdXNE+Y8y3Kgryj1hrNwWzLqd5pFsVAOA0Fx8fr8zMTGVlZSklJcXtcoBqJTo6WvHx8W6XcUyOBPnibjAlXWEaFz8ONMa8Xfw81Vo71olrVVc0XgAAvCYkJEQtWrRQWlqaMjIylJub65lBfkAgGGMUGRmp2NhYxcfHKyQk6BM8nhCnWuR7S7rhqG1ti38kaZukGh3kAQDwopCQECUkJCghIcHtUgCcIEe+ZlhrJ1hrzTF+WlfhHEOLj/V0txoAAAAgGKr3/QIAAAAAFSLIBwC9CwEAABBoBHmHMNYVAAAAwUSQBwAAADyIIA8AAAB4EEEeAAAA8CCCfACwmAYAAAACjSDvEMPSrgAAAAgigjwAAADgQQR5AAAAwIMI8gAAAIAHEeQDgKGuAAAACDSCvEMY6goAAIBgIsgDAAAAHkSQBwAAADyIIA8AAAB4EEE+AFjYFQAAAIFGkHcIC7sCAAAgmAjyAAAAgAcR5AEAAAAPIsgDAAAAHkSQBwAAADyIIA8AAAB4EEHeIYZpawAAABBEBHkAAADAgwjyAAAAgAcR5AEAAAAPIsgHiLXW7RIAAABQgxHkAQAAAA8iyAMAAAAeRJAHAAAAPIggDwAAAHgQQT5AGOsKAACAQCLIO4jFXQEAABAsBHkAAADAgwjyAAAAgAcR5AEAAAAPIsgHCGNdAQAAEEgEeQcx1hUAAADBQpAHAAAAPIggDwAAAHiQI0HeGHOlMeYlY8x8Y8xhY4w1xrxTybEdjDEPGmO+McbsMMbkGWP2GmM+NcYMc6IeAAAAoKYLc+g84yT1knREUoqkzsc49i+SfilpraTPJKVJ6iRplKRRxpi7rLUvOlQXAAAAUCM5FeTvUVGA3yTpHElzjnHsF5KesdauLL3RGHOOpK8kPWeMmWKt3e1Qba6w1orhrwAAAAgUR7rWWGvnWGs32qL0erxj3z46xBdvnyvpW0kRkgY5UVewGUNwBwAAQHBUt8Gu+cWPBa5WAQAAAFRzTnWtOWXGmFaSzpOUJWleFd+TVMmuY/XRBwAAADyvWgR5Y0ykpHclRUp6wFp70OWSAAAAgGrN9SBvjAmV9B9JgyV9IOmvVX2vtTaxknMmSerjSIEn6biDBQAAAIBT4Gof+eIQ/46kMZI+lHRdVQbMVlcMdQUAAECwuBbkjTFhkv4r6WpJ70n6lbWWQa4AAABAFbjStcYYE6GiFvhLJf1b0o3WWp8btQAAAABeFPQW+eKBrZ+oKMT/S4R4AAAA4IQ50iJvjBktaXTxy8bFjwONMW8XP0+11o4tfv6apBGSUiXtlPSnChZS+tZa+60TtbnFuz39AQAA4AVOda3pLemGo7a1Lf6RpG2SSoJ8m+LHBEl/OsY5v3WotqBhYVcAAAAEiyNB3lo7QdKEKh471IlrAgAAAKczV6efBAAAAHByCPIAAACABxHkA8SytisAAAACiCDvIMPargAAAAgSgjwAAADgQQR5AAAAwIMI8gAAAIAHEeQDhJVdAQAAEEgEeScx1hUAAABBQpAHAAAAPIggDwAAAHgQQR4AAADwIII8AAAA4EEEeQAAAMCDCPIOYtIaAAAABAtBHgAAAPAggjwAAADgQQR5AAAAwIMI8gFirdsVAAAAoCYjyDvIMNoVAAAAQUKQBwAAADyIIA8AAAB4EEEeAAAA8CCCfIBYMdoVAAAAgUOQd5BhbVcAAAAECUEeAAAA8CCCPAAAAOBBBHkAAADAgwjyAcLKrgAAAAgkgryDWNkVAAAAwUKQBwAAADyIIA8AAAB4EEEeAAAA8CCCPAAAAOBBBPkAYdIaAAAABBJB3kFMWgMAAIBgIcgDAAAAHkSQBwAAADyIIA8AAAB4EEE+QKxluCsAAAAChyDvIGMY7goAAIDgcCTIG2OuNMa8ZIyZb4w5bIyxxph3jvOeQcaYz4wxacaYLGPMKmPM3caYUCdqAgAAAGqyMIfOM05SL0lHJKVI6nysg40xl0r6WFKOpA8kpUm6RNLfJQ2WNMahugAAAIAayamuNfdI6iipjqQ/HOtAY0wdSW9KKpQ01Fp7k7X2fkm9JS2SdKUx5mqH6gIAAABqJEeCvLV2jrV2o63aCM8rJTWQ9L61dnmpc+SoqGVfOs6XAS9gqCsAAAACyY3BrucWP35Rwb55krIkDTLGRAavJGcw1BUAAADB4lQf+RPRqfhxw9E7rLUFxpitkrpJaitp3bFOZIxJqmTXMfvoAwAAAF7nRot8XPFjeiX7S7bXDUItAAAAgCe50SJ/PCU9VI7bzdxam1jhCYpa6vs4WRQAAABQnbjRIl/S4h5Xyf46Rx3nSSzsCgAAgEByI8ivL37sePQOY0yYpDaSCiRtCWZRjmC0KwAAAILEjSD/TfHjLyrYd7akaEkLrbW5wSsJAAAA8BY3gvxHklIlXW2M6Vuy0RhTS9ITxS9fdaEuAAAAwDMcGexqjBktaXTxy8bFjwONMW8XP0+11o6VJGvtYWPMLSoK9N8aY96XlCZplIqmpvxI0gdO1AUAAADUVE7NWtNb0g1HbWtb/CNJ2ySNLdlhrZ1mjDlH0qOSrpBUS9ImSfdKerGKK8RWb97/DQAAAFCNORLkrbUTJE04wfd8J2mEE9evLhjrCgAAgGBxo488AAAAgFNEkAcAAAA8iCAPAAAAeBBBHgAAAPAggnyAWKatAQAAQAAR5B1kDPPWAAAAIDgI8gAAAIAHEeQBAAAADyLIAwAAAB5EkA8Qy1hXAAAABBBB3kGMdQUAAECwEOQBAAAADyLIAwAAAB5EkAcAAAA8iCAfIIx1BQAAQCAR5B2Unp3vdgkAAAA4TRDkHVR6yslVKYfcKwQAAAA1HkE+QP6zaJvbJQAAAKAGI8gDAAAAHkSQBwAAADyIIB8gzFoDAACAQCLIAwAAAB5EkAcAAAA8iCAfIMbtAgAAAFCjEeQBAAAADyLIBwiDXQEAABBIBHkAAADAgwjyAAAAgAcR5AEAAAAPIsgDAAAAHkSQBwAAADyIIB8g1jJvDQAAAAKHIA8AAAB4EEE+QIxhbVcAAAAEDkEeAAAA8CCCfIDQRx4AAACBRJAHAAAAPIggDwAAAHgQQT5AGOwKAACAQCLIBwgxHgAAAIFEkAcAAAA8yNUgb4wZaYz50hiTYozJNsZsMcZMMcYMdLMuAAAAoLpzLcgbY56RNENSH0lfSHpB0gpJl0r6zhhznVu1AQAAANVdmBsXNcY0ljRW0l5JPa21+0rtGybpG0mPS3rHjfqcwFhXAAAABJJbLfKtiq+9pHSIlyRr7RxJGZIauFGYc0jyAAAACBy3gvxGSXmS+htjEkrvMMacLSlW0tduFAYAAAB4gStda6y1acaYByX9TdJaY8w0SQcktZM0StJXkn7nRm0AAACAF7gS5CXJWvsPY0yypLck3VJq1yZJbx/d5aYixpikSnZ1PvUKAQAAgOrLzVlrHpD0kaS3VdQSHyMpUdIWSe8aY551qzYnMNgVAAAAgeTWrDVDJT0j6RNr7b2ldq0wxlwmaYOk+4wxr1lrt1R2HmttYiXnT1LRtJYAAABAjeRWi/zFxY9zjt5hrc2StFRFtZ0RzKIAAAAAr3AryEcWP1Y2xWTJ9rwg1AIAAAB4jltBfn7x463GmGaldxhjLpI0WFKOpIXBLgwAAADwArdmrflIRfPEny9pnTHmE0l7JHVRUbcbI+kha+0Bl+o7ZYx1BQAAQCC5NY+8zxgzQtLtkq6WdJmkaElpkj6T9KK19ks3agMAAAC8wM155PMl/aP4p8Zh+kkAAAAEkmvzyAMAAAA4eQR5AAAAwIMI8gESGRbqdgkAAACowQjyARJCH3kAAAAEEEE+QEIY7QoAAIAAIsgDAAAAHkSQBwAAADyIIB8o9KwBAABAABHkAQAAAA8iyAeIoUkeAAAAAUSQD5CIMII8AAAAAocgHyA9m9d1uwQAAADUYAR5B12Z2NztEgAAAHCaIMg7KIzlXAEAABAkBHkAAADAgwjyAWKt2xUAAACgJiPIO8jQswYAAABBQpAPECua5AEAABA4BHlH0SQPAACA4CDIAwAAAB5EkA8QBrsCAAAgkAjyDmKwKwAAAIKFIB8gNMgDAAAgkAjyDqJBHgAAAMFCkAcAAAA8iCAfKIx2BQAAQAAR5B3EYFcAAAAEC0E+QGiPBwAAQCAR5B1kGO4KAACAICHIAwAAAB5EkA8QxroCAAAgkAjyDmKwKwAAAIKFIB8gliZ5AAAABBBB3kE0yAMAACBYCPIAAACABxHkA4SONQAAAAgkgryDDKNdAQAAECQE+QBhrCsAAAACiSAPAAAAeBBBHgAAAPAggnyA0LMGAAAAgUSQdxBjXQEAABAsrgd5Y8xZxpiPjTG7jTG5xY9fGmNGuF3bqWBlVwAAAARSmJsXN8aMk/QXSamSZkjaLSlB0hmShkr6zLXiToJhbVcAAAAEiWtB3hgzRkUh/mtJl1trM47aH+5KYadgX0aO//nu9JxjHAkAAACcGle61hhjQiQ9IylL0q+ODvGSZK3ND3php2jGqt3+5/9asNXFSgAAAFDTudUiP0hSG0kfSTpojBkpqbukHElLrbWLXKoLAAAA8AS3gny/4se9klZI6lF6pzFmnqQrrbX7j3USY0xSJbs6n3KFAAAAQDXm1qw1DYsffy8pStL5kmJV1Co/S9LZkqa4UxoAAABQ/bnVIh9a/GhU1PL+Q/HrNcaYyyRtkHSOMWbgsbrZWGsTK9pe3FLfx8mCAQAAgOrErRb5g8WPW0qFeEmStTZbRa3yktQ/qFUBAAAAHuFWkF9f/Hiokv0lQT8qCLUAAAAAnuNWkJ8nqUBSB2NMRAX7uxc/JgetIgAAAMBDXAny1tpUSR9IipP0p9L7jDEXSBouKV3SF8GvDgAAAKj+XFvZVdK9kgZIetQYc7akpZJaSbpMUqGkW6y1lXW9AQAAAE5rrgV5a+0+Y8wASeNUFN7PlJQhaaakp621i92qDQAAAKju3GyRl7U2TUUt8/e6WQcAAADgNW4NdgUAAABwCgjyAAAAgAcR5AEAAAAPIsg76MKujfzP+7aq52IlAAAAqOkI8g46q0OC/3mnxrEuVgIAAICajiDvJGP8T62LZQAAAKDmI8gHyMxVu90uAQAAADUYQd5BX63d63+enp3vYiUAAACo6QjyDtp5MMvtEgAAAHCaIMg7yJTqIw8AAAAEEkHeQSHkeAAAAAQJQd5BIbTIAwAAIEgI8g6iaw0AAACChSDvIGI8AAAAgoUg7yAa5AEAABAsBHkH0UceAAAAwUKQdxCz1gAAACBYCPJOokUeAAAAQUKQdxAt8gAAAAgWgryDyPEAAAAIFoK8g5hHHgAAAMFCkHcQMR4AAADBQpB3EA3yAAAACBaCvIPoWgMAAIBgIcg7iBgPAACAYCHIO4iVXQEAABAsBHkHkeMBAAAQLAR5BxHkAQAAECwEeQfRtQYAAADBQpAHAAAAPIgg7yBa5AEAABAsBHkHhZDjAQAAECQEeQexIBQAAACChSAPAAAAeBBBHgAAAPAggryD6FgDAACAYCHIAwAAAB5EkHcQY10BAAAQLAR5B1nrdgUAAAA4XRDkHfS7c9q5XQIAAABOEwR5B7WuH+1/Hh0R6mIlAAAAqOmqTZA3xlxvjLHFPze7Xc9JKdVHniAPAACAQKoWQd4Y00LSS5KOuF3LqTClkjz95QEAABBIrgd5Y4yRNEnSAUmvuVzOKSk9aw05HgAAAIHkepCXdKekcyXdKCnT5VpOSenZJy1N8gAAAAggV4O8MaaLpImSXrDWznOzFieYUk3yxHgAAAAEUphbFzbGhEn6j6Ttkh45yXMkVbKr88nWdSrKtsi7UQEAAABOF64FeUl/knSGpCHW2mwX63BMmT7yJHkAAAAEkCtB3hjTX0Wt8M9baxed7HmstYmVnD9JUp+TPe/JKj1rzeGcgmBfHgAAAKeRoPeRL9WlZoOk8cG+fkCZsi8LfbTKAwAAIDDcGOxaW1JHSV0k5ZRaBMpKeqz4mDeLt/3DhfpOmjkqyP+QcsidQgAAAFDjudG1JlfSvyrZ10dF/eYXSFov6aS73bjhqByvg5l5rtQBAACAmi/oQb54YOvNFe0zxkxQUZCfbK39ZzDrcoI5qkmenjUAAAAIlOqwIFSNcfRMNcxcAwAAgEAhyDvo6NhOjAcAAECgVKsgb62dYK01XuxWAwAAAARTtQryNQ09awAAABAoBHkAAADAgwjyDooODz1qC03yAAAACAyCvIPCQvlzAgAAIDjhA5BQAAAgAElEQVRIngFEH3kAAAAECkEeAAAA8CCCfADRIA8AAIBAIcgDAAAAHkSQD6D9GblulwAAAIAaiiAfQP9dut3tEgAAAFBDEeQDKDTEuF0CAAAAaiiCfAAR5AEAABAoBPkAalEv2u0SAAAAUEMR5AOoZ/M4t0sAAABADUWQDyDmkQcAAECgEOQDqH2D2m6XAAAAgBqKIO+wwe3r+5/TIg8AAIBAIcg7LDoizP88OTXTxUoAAABQkxHkHfbV2r3+509+ts7FSgAAAFCTEeQBAAAADyLIAwAAAB5EkAcAAAA8iCAPAAAAeBBBHgAAAPAggjwAAADgQQR5AAAAwIMI8gAAAIAHEeQBAAAADyLIAwAAAB5EkAcAAAA8iCAPAAAAeBBBHgAAAPAggjwAAADgQQR5AAAAwIMI8gAAAIAHEeQDzFrrdgkAAACogQjyATZvY6rbJQAAAKAGIsgH2M2Tl7ldAgAAAGoggnyA5RfStQYAAADOI8g7bHTvpm6XAAAAgNMAQd5h4y/uWm7b3sM5LlQCAACAmsyVIG+MqW+MudkY84kxZpMxJtsYk26MWWCMuckY49kvGPVrR5bb9uf/rXGhEgAAANRkYS5dd4ykVyXtljRH0nZJjSRdLumfki4yxoyxNWTuxt3ptMgDAADAWW4F+Q2SRkmaaa31lWw0xjwiaamkK1QU6j92pzxnHczMc7sEAAAA1DCudGGx1n5jrf1f6RBfvH2PpNeKXw4NemEBknwgy+0SAAAAUMNUx77o+cWPBa5WAQAAAFRjbnWtqZAxJkzSr4tfflGF45Mq2dXZsaIAAACAaqi6tchPlNRd0mfW2lluF+OkQl+NGLcLAACAaqLaBHljzJ2S7pP0k6Trq/Iea21iRT/F56hWfvefJMI8AAAAHFMtgrwx5nZJL0haK2mYtTbN5ZIc9/W6vfrXgi1ulwEAAIAawvUgb4y5W9LLklarKMTvcbmkUzbpN/0q3P7UZ9XuRgEAAAA8ytUgb4x5UNLfJX2vohC/z816nNKtWR23SwAAAEAN51qQN8aMV9Hg1iRJ51lrU92qxWlhIZX/WUe9vEALN6UqadtB5Rf6Kj0OAAAAOBZXpp80xtwg6XFJhZLmS7rTGHP0YcnW2reDXJojQsv/Ln6rUtL1q38ukSSd1SFB/7lpQLDKAgAAQA3i1jzybYofQyXdXckxcyW9HZRqHOazVZudZv7GVOUX+hQe6vpQBQAAAHiMKwnSWjvBWmuO8zPUjdqcEBNZ9e9HTEkJAACAk1GtVnatKSLCqv79qPP4L9Q0rpbqRkfo77/srU6NYys8LiMnX7G1wuXzWSUfyFSbhBhV0B0JAAAApwn6dARI/zbxVT52V3qO1u4+rJsmL9OGvRnKyS8ss/+l2RvV689f6o73VuiGSUt17vNz9ei01U6XLJ/P6n8/7NKU5TsYiAsAAFDN0SIfICEn0ViecjBbF/59nqSigbD3D++kns3r6vmvNkiSZqza7T/2vSXbNW5kF70we6PCQoz+eG4H1QoPrfC8e9Jz9NRn69QwNlIPj+ii0EqKm7N+n/7435WSpAKf1TX9W574LwEAAICgIMgHyEXdm2jxlpNfoHb+xlQt2Zqm7x48t9Jjuv5plv95ZFioLunVVJ+s3Kk+LetqaKeGkqQdaVk669k5/uNWpaRr8m/769v1+9S3dbwaxEb699387+X+5w9P/THgQX7z/iN66ONVahwXpf5t4rVy+0HdPqy92jWo7T9m8sJkfbh8h24b2l4jezYJaD0AAABeQpAPkAFtq961pjJ5BT71e/LrKh37t6826G/FLfeS9N1D56pZ3Sg98smPZY5bmpymuz9YqVlr9kqSVo6/QLXCQxUVEaqjJ9vZezhHj37yo0JDjO65oKM6Ny670JW1VsYYpWfn66a3l2n5toO69ey2uveCjpXeHSjt9/9J0sZ9RyQd1P9+2CVJStp2UHPvHyZJSs/O12PT10iSbn9vhUb2HFnheXw+q7kb9yssxGhI+4Sgjx2wxX+4xVvSFBMZqp7N6wb1+gAA4PREkA8Qn8tdzAdP/EYz/jhE8zeWX2erJMRL0hl/+UqS9MGtZ5Y7bsBTs/3Pk7Yd1NJHzteiLQfUrkFtHcnN163/TlLygUyVnnjnjXlbVC86Qn8Y2u6Y9RUU+opDfFnbDmRp0eYDmr1urwa3Tzju75mRk6+v1+3VPR/8IEl656YBGtLh+O+rTHJqphrWiVR0RNU+Gp9+v1OP/2+tJOlAZl7RttsHq1cLwjwAAAgsY6s457mXGGOS+vTp0ycpKcm1GlbvTNfFLy1w7fqBVj8mwh9cK7L+iV8oMuznVvnsvEJtST2ikS8uUOfGsTqQmaf9GbkndM3kiUUt8nkFPoWHGv20J0NXvLpQWXk/Dw6OjgjV2sd/Ue69e9Jz9MDHq1Q7MlTPXdlLa3Yd1uSFyRrVu6mGd2ssSfrv0u16eOqPqh8ToXkPDKvSNKKtH5pZblvbhBh9M3aovl67Vyt3HNQNA1urYZ1aJ/S7AgAAb0hMTNSKFStWWGsTg31tWuQDpKqLQnnVsUK8JJ351Gzde0FHjf90Tbl9P+3JOOnrLktO05jXFlW632etrLXKyivU1+v2KjuvUJf3aa6Hp67SvA37JUkt6kXr9XlbJEkzf9ytpY+ep4axtfTw1KJuSAcy8zTmtUW67sxW+kX3xoqPifCf/+u1e7XjYJau6ttC2UfNLlRiS2qmth/I8o85WL3zsCb/tv9xf7d5G/ZrwaZUXTeglVrWj67aH+QohT6rA0dyq9UXh8M5+fpqzV4NaBuv7LxC5RX61K1pnNtlAQDgeQT5AGlR7+SCWE1xMCu/whB/Kipq/T5aTr5PbR7+rMy2fJ/VnPX7/a+nF/fHL9H/ydk62trdh/XIJz/qb19t0EMXdVbL+Ght2JuhccXTfi7dmqbPV++ptI4Plm/3P5+7YX+lx5U4cCRXv35rqaSiQD/1tkE6cCRPLeKjlV/o06qUdB04kquB7eortlZ4uffvSc/RW99t1RvFX1D+PKqbbhjUusJr/bDjkGav26srEpurVf2Y49ZWWk5+oSLDQk5oHMJDH6/SZz+W/Vv9+7f9dXbHBid07bwCn75dv09dmtRRi/jgfr72Hs7Rkq1pOrdzQ9U+gQXfAAAIJLrWBNC0lTt19wffu1oDqoenL++hy85opojQEIWEGB3KytNj09eooNBqwqhumrthv8ZO+cF/fFR4qLLzC3XneR304uyN/u19WtbVrWe31btLtmtM3xYa1aupJOnKVxdq+baDZa7564Gt1Ld1vPYdztHV/VsqJiJUuQU+dR7/hf+YHydcWOEXA2utDmTmKaF2pHakZWnC9DXasC9DqRl5ahEfpel3DKnSgGap4i9gxkhbn6548HJlJn7+k16bu1kxEaFa/Mh5FdYdCIU+q7OfnaOdh7I1okdj/d+1Qb9zCgCoxtzsWkOQD7Aej81SRm6B22WgGhnVq2m5uwIna+FD5yorr1Dn/21ulY5/9+YBuvafS8ps+9cNfXVOxwby2Z9XJb558jJ9vW6f2iTEaGtqZoXn2vLUCIVUsCbB/I37Nem7ZA3v1kidGtfR6Fe+q/D9fxjaTokt6+nczg310NRV2rw/U09e1r3c7Eg7D2XrzXlb9PbCZP+2R0Z01q1nH3tAtVN+2HFIl5b6HUrGaqBi1lp9v+OQWsZHq37tyOO/AQA8jj7yNdjjo7v5Z1QBpPJde07FoInfnNDxR4d4Sbpp8s/rBwxoE68lW39e/6CyEC9Jn6zcqSsSm8vns1q7+7BufHtZmQHM3/y0T20TKu+68+q3myVJd57bXh8uT5Ek3fDWUi155HwV+qy2HchUm4QY3fXfleXuNhQUT5WUnJqpZ774SR0bxeru8zvIGKO8Ap//C8kb8zZr7ob9uveCTtq4N0Pp2fmqExWux6avUV6BT9cOaKk/j+qmsNCfF7ku9Fl989M+1Y0OV7/W8eXGu/zj6w3q1aKuMnML1KlRrDo0iq30d6yMtVb5hdZfp1Q0jWpeoe+4dzqstZq7Yb/W78lQysFsDW6foF90b3zCNThtw94MZeYWKGnbQT0xc51iIkK18OHzFBcVnDsnp8rns3pi5jrtPJSlcSO7Br37FoJn9c50xUWF898YNQIt8gFW6LP6cPkObd53RP9csNXtcoBq7/1bz9TVbyw+5jFjL+yoZvWiynxJvnFwazWvF62/zCiaDvRE73wkTxypQp/VHe+t8I9/mH7HYKUczNZt76447vvn3T/suIOUc/IL9bv/JPnHTXz4u4Hq17qeMvMKNerlBdqyP1MxEaH68PcD1bhOLYWFhpQLwh8u26EHPl5VZtvk3/ZXp0axiokMVe3IMBlj9P2OQ0o5mKULuzYu84WhKrLyCo45BWt6dr7emLdZ9WMidePg1lqz63CFs3TdPqyd7h/e+YSufTgnX4s2H9CgSsaDBMqHy3fogY+K/q6Jrerp4z8MCtq1vapkLREvmblqt25/b4WMkb65b6jaHKOxAagqutY4rDoF+RIHjuQq8YmqLe4EwJs+/N1AfbJyp4Z1aqALuzXW7vRsvTh7k9o1iNHNZ7XV37/aoBdKjXmQpO7N6qhDw1h9snJnhee8sGsjvXpdokJDjEa9vECrUtKPWUP/1vF69sqeGvrXbyVJDWMjtfTR8yVJaZl5em7Wet16dtsyAcZaq4mf/+SfzSk0xGh072Z6/qpeFV7jyZlr9eb8ooaJetHhysn3VTiL028GtdaEUd0kFa0yfft7KxQVHqo3ft1XcVHhys4rVEZOvhrWqaUV2w9q6ooUvbtku6wtujv0we8GHvN3PdpPew7rgY9WqUV8tF74ZW/NWb9f//ftJq3cfkjSsdd4uHHS0jKD4ide3kMXdG0U8O5BPp9VboFPURHl78SUDsq7DmXr/WU7NLhdfQ1oWz+gNVXmx5R0PfDxKrVrEKMezeL0ypxNumFQa913YSdX6iktt6Do3196dr4axlY+a1fpMTuD2tXXe7eUX0MFRVNGV/RvEhUjyDusOgZ5Sbrun0u0YFP5BZoA1HwvXXOG/vjflUG5Vq3wEOXkl12Vrl/relqW/HMXpdV/Hq6731+pr9ftq/Q8o3s31T+uPkOSdCS3QK/PLeqqdLwvE6W1bRCjQ1n5Sis1ZW14qNHtw9rrH19vPMY7i+6SWGt1y7+T9PW6vfrbVb10eZ/m+jElXV+u3aPL+zRXozqRGv3Kd9qwt+wCczERocrMK//lInniSB3JLdDUFSnak56jEGO0cHOqVhSH/aOd3bGBftW/pfq1rqfoiLBKA/emfUfUtkFthYYYrdmVrt9MKupqdsew9rrngo4KLR5PciS3QJFhIQoPDVF2XqEueXmBdh/K1qvXJersjg2UV+DT63M3a8GmVH83twaxkWW6rY3o0Vi/O7uderWoq7kb9uuGt5ZqwiVd9ZvBbfzHpGfnV9itKSe/UPszchUeGqIfUg7pnI4NKu3ONXvdXj312TrViQrXWzf00+BnvimzbkeJtY8PP+YdnBPpNiap0lb+g5l5ql0rTOGlusKV/vdRYvzFXXXTkDYVnaJMkO/ZPE7T7xjiP48bdxeSUzP16fe7dGG3RurSpI6+25SqWWv2aExiCzWqE+mfSrjQZ7U/I1eN45yfWvidxdv0vx926Y5z2+usDg304uyNemH2Ro1JbK7zuzTSsm1pumFgazWtG+X4tU9ETn7R9MVJyQfVIDZS3ZtVn2mMCfIOq65B3lpbbmpEAEDwPH5pN329bp9/XYkTNfHyHhp9RjNt2JuhPek5OrdzQ42btlrvL9vhb+E9eqam1vWjNaxzQ2XnFWrqip2qGx2uns3rlgmfknRJr6aatXqP8gqPvzR4iJHWPv6LMrNQvXZdoga3r68eE76UJF3ep5meuqyH3luyXXWiwjWoXf0Kx9U8c0UPXdC1sepFh8sYo7TMPL2zeJv+9tWGKv1NfjWgpXYezNYjI7qoU+Ofx4ykZ+eroNDnvxvdp2Vd1Y2O0KMjuygyLESb9h3RkPYJCgsN0bYDmTrnuW8lSc+P6aUrEpvrv0u3Kye/UL/s10ILNqbqjvdWKqF2hJ6/qremLN+h4d0ba+aq3RV2oUueOFIpB7M0b0OqLujaSHWjw5VX4FO3x2aVOe6py3roq7V7tHb3Yd16djttO5CpoZ0a6NzOjcqdc+2uw0raflCjejZVXHS4klMz9cHyHWpcp5Z+2a+FaoWHKr/Qp1lr9ujJmet0Ufcm+tMlXZVyMEv3fPC9IsJC9NI1fZSWmacGsZGavDD5mH/jECP9/Ze9dXHPpmr3SFF2aBJXS4sePu+4/032Hs7RzkPZ+tf8rTqUnadnruip5hVMi70/I1f9nvy5t0DyxJEVzjTWv3W83rtlgIwx/i+lJ2N/Rq4+WLZdfVrV06B2x16FvaDQp52HstWqfoz2Hs7RRS/ML9Mg8PEfBimxVT1J7nfzIsg7rLoGeUnq+OjnZf5H+qPfD9SY1xepBv5nAIDTzlu/6avfvr38+Ac64P7hnfTcrPWOnvP2Ye30ypzNJ/3+iNAQ1a8dod3pOVV+T99W9coNaC+td4u6+n5HxXdMKnNFn+b6eEXKCb2ntCZxtfTSNWcoJ9+nXi3iZIzRwKdmKyO3QFf0aa7nr+qlgU/P9v+ejevU0m3D2ulvX23Qoax8/3lG9miimT/uPuk6pKIvG4988qP/9YIHh2nF9kO6s/gO32d3nqWuTX+e7Wt3erbOfnaO8gvLBovXr0/Uiu0H1b1pnC7u2URLtqbpgY9WaXtalv+Yy/s009QVFXfzK1nR/enLe+ia/i3926212pWeo6teW6QOjWorMixEMZFhemJ0d/+dmpz8QtUKD9VVry3S0uSiO02f3DZIsbXC9cf/rtTo3k11y1lttXLHId08eZkSakdq7+EcHc4p0OjeTbV5f6Z+3Fn+TuCmJy/Sbycv17wN+/WbQa318IjOZVaVDxaCvMOqc5DfkZalK15dKCvpxavP0MB29ZVf6NPMVbuZcx4AAA/4zaDWZabEDaTh3Rpp1pq9xzwmoXak2jWIUURYiOZvDE4X3hbxUdqRln3MY2JrhSkjp6Dc84qczJfgns3jynT1+/057fTQRSc2wN4JBHmHVecgfyxXvLpQScdolQAAAEDl3Fjrw80gf2JzkiGg/u/aPppwSVfNvX+onrqsR5l9Cx4cVu74Xw9spS1PjdC7Nw9Qm4QY9WtdT/93bR9teWqENj55kTpVcX7r6IhQPTKis341oKW/v5kTmhw1KIel7QEAQCD5fDWvgfpYSFbVSKM6tfyzDjStG6XwUKPlyQf12yFtyg1SmXRjPw3r1FCSNLh9guaMHVpmf4iMZt1ztiZMX+O//RcVHqpnr+ypqStSFBcVrolX9Cw3i0DJYjPf/LRP/160zb89PNToT5d005jE5goLKRrsMuyv3yr5QFHfuhsGtlKP5nU1dkrRvN5v/rqvLujaSP9dul3Pf7lBv+zXXHee10Gz1uzVo5/8qOy8Qv16YGuN7NlYufk+dWgUW2bAzbHcclYbtYiP1hvztiihdtE81ne9T7ckAABOd6/O3azbh7V3u4ygoWuNh5SE8hbxUfp27LBTGjleFVOW79CCTan6w9B2al4vulyLenp2vqauSFHP5nFKbBUvSdq4N0NZeYVl5mo+ejR5bkGh8gp85RZ76Tlhlg5X0n9uyu8HatJ3WzWoXYKuO7NVuf1//2qDFm85oFb1ozWkQwP/IKCqOL9LQ13Rp7kKfFaHsvM1ftrqKr8XAABUL8HuXkMfeYfV1CDv81klbT+oLk3q1MhuKulZ+VqwKVVDOiSoVniICgqtYk7y91yenKYrX1tUbnvzelH65r6h6v34l8rKK6xwvuGKpt46Wt9W9fTh7waq7SNlpxPt1aKufjhqdoVrB7TUhr0ZGn9xV6UeyQ3ajBYAAJyOTqcgX/PSYA0WEmLUr3W822UETFx0uEb2bOJ/fSrfVfq2jvd/kO+f8oOmJBVNQ3bj4DaKCAvR2sd/Uel7//7LXho7ZZUKK+ln9+3YoWoZH62QEKMZfxyiq99YrLBQo//dMUQt4qPLfRF48qjxDi9dc4bm/LRPfzyvg4YVr74pSe0b1tamfWUXtTnaQxd11uItB7QnPUdbUjOVV+BT07ha2nUCU72VaNsgRlv2Z1a6v3m9KPVqXveUp04DAACBQZBHjffIiC4KCzWKrRWu6yvolnO0y85orrM6NFDdqHClZ+crO79QZz87Rz4rPXxRZ7UutbR992ZxWvLIeQoNMf7xBqWnw+rerE6581/Sq6ku6dW03PYXru6tyLBQnf+3uf5t7948QGOn/KDdxQvP/P6cdvr9Oe0kFc3LGxlWNF795W826fnihUXaNYhRlyZ1NGPVsQP4lN8N1BWvLvSPczjax38YpIaxkdr/eq5/3t+GsZGykn+VyYUPnasmcbX8XacKfVYzVu1S0raD2nkwW7N/2qeI0BAltqqnge3qV3mBmZK/x85D2Xr2C2fmyT6ZuagBAKjO6FoDVMHqnenakZal87s2KrM8eEW2pmbq1n8vV3RkmCbf2E91oyMqPTZp20E9N+snndWhgW4f1l7WWl352iIlbTuo685sqSdG99Ce9Bwt3Jyq8zo3Ulx0+SXXS8zbsF+LtxzQdWe2UnxMhP46a70KfFZXFq+Q2L9NvC7t3Uyb9h1R07q1FB0RpqVb03TV60VdkJ66rIcu79NM8zemqlX9aHVsVHaFxo17M9ShUaziosK1NTVTDWIjq9TFK7/Q5/+b5Rf6dOnL32nT/iN6fkwvDW6foDfmbVFsrTBd0LWRCn1WtSPD1CL+58Hdr367WV+u3aN7L+ioszo0kPTzUu5DnpmjnYfKz2N8+RnNNLBdfd3/0SpJPy+Y8vmPuzVu2mqN6t1Uk75LPm7tUtGCI+0f/dz/ul2DGI3s2VQvzt5YpfcDAIKnZ/M4Tb9jSFCvSR95hxHk4bZTWS66oNCnzfsz1bFR7YAvOW2t1Zz1+5SRU6ARPZoc90uKU9fMzCt0ZJxHQaFPHyWlKCoitMzMRSVLvO9Jz1F8TIQiwsr/XpMXJuv1uZvVt3W8Hrukq38Z+dISW9XTx38YpFfmbNJzs9areb0ozRk7VOGhIWW6UCXUjlCTuCilZ+frkRGddWHXxsr3+WSttGjzAd349jL/sV/cfZb+PH2tFm054N92db8Wen/ZDv/rrU+PUFpmnmat2etf0bF5vSilHPz5S8td53XQC1X4MvHR7wfqyc/WaeX2iu9GvHB1b721YKsu6dVUr83drNQjeeWOWfLIecrOK9Tr8zYrLipCr82tfOXPhQ+dq0ETvzluXVXRMDZS+4rv/lTkrA4JZRa/ubR3UyWnZuqHlPIrQJ6oF68544QGzQOoHv46ppeuTGwe1GsS5B1GkAdOP2NeW6hlyUULqi199Dw1jK11nHeUtTw5Tdf+c4kiw0L0+KXdtTU1U7/s10JN60ZJKpqRqUV8tL8L1Rerd+v376xQ7cgw/fDYhcecRWr0K9/p+x2H9ItujfXa9Yk6nJOv856fq0NZefq/axPVt1U9DXnmG2XmFR7z/4TyC32au36/MnLzdXHPplqwKVU3TlqmYZ0aqGV8tN5bul2D2yfo2/X7/e+Z/8AwFfqsnvxsndomxOj6ga20YvshTV6YrCsTm5dZan1/Rq4Wbk7V0E4NtTU1U3Wjwst0JSuxZle6Hv/fWnVvFqf7h3fSkGe+UeqRPN11Xgfdc0FH/7kKfVY/7Tms/yzapjnr9+nu8zvq/C6NNOLF+ZKkK/o011/H9NS/F23TY9PXSJKevbKnrurbosz1znr2mwpXkEyeOFLjpv2odxZvL6rrz8MVExmm1TvT9c1P+1QrPESTvkvW7iqMIZl55xCNfHGBJCkmIlRrHv+F9mXkqP+Tsyt9z2OXdNXOg9ka2K6+bppcdhD7/cM76blZx+4WdlaHBF3Tv6Vue3dFpcf8Yej/t3fnYVJU5x7Hv+8wMOygCCigggiKggRxAyKoGKOiEU30Rm9wy+aNcc1z3XI1mBglbjEabxKNCy5J3CE3bigEUYgRV0CHTRhkkwFGlllhZs7945wemp7uWXpmehl+n+fpp2aqzqk+/XZV11tVp6oG8ofZiXecIn5/wQh++pfEOx65OUZlPffXvnh0f9rl5vDQnBV079iWLaU7E5YddVCP3XZGk9W9Y1seueho5hcUMeXVxXHLxN5E4OELj+KeGUtY/OX2Jr+/tD4zrhm72xnlVFAi38yUyIvsedZtKePpf6/i2AE9GDu4Z1Lz2Fq6k3a5OXRo16b+wjT8zEv5zio+Xr2FkQfuVXPWo6KyiuLySnp0zqt5743F5Rzcq3EboOg2VFRWkZfbhlcWrueu15dw5vA+XBsS65a0tWwnn28sZsT+3RPGo6SisuYuVK8t+pJlG7YzadSBNV3PFq7ZyqaSCsYO6llrp6ikopKXF67nkN5dmPTIv9lWXsmgXp1549pxAOyorI571iXCOcfywmI2Fe/g/Iff3W2aGTxy0VGcdGhvFq3dyquL1nP2iL4130NkJwzgyAO6M/lbh3NEv+613uPzjcWMv8df33LJmP784szDuf75BTzzvj/TcvvZw6isruaW6X6HZeqlxzBu8K6uYsUVlXRp35Zbpi+qeYbHaUP35fcXHMkz81cz5dV8fj5hCI++U8CqohKuOGnQbjsKBVMmsGjtVp56dxXjh/TGgCUbtlO4rZwrxg+ijRkL1m5l9MAelO2sYt7yTQzu3YV/rdjM8H7dad+2DQf36rzbZ4o+6/TH743ksqf8NvWB80dw5vA+vPDBGq5/YQEd27Xhx+MG1rRn8pmH8Ub+BuYu94l+n27tufH0IczM38C0j9cB0LtrHjeeNoSJI/rWfIeD/+dV4imYMoHrnv+EZ99fwyzWehkAABN9SURBVE2nH8qPxvrrhLaU7uCxuQX89b0vKNxewYE9fJe8VQmu+3ng/BHk5ebwoyf955h03IH8auJQpn+8Nu6zSM4c3odzjuzLF5tL2bi9glOH7ssZD7wTd94Rz102inOj7ph2z7nDGTu4Jx+sKuKypxLvsN173nCuffaTOucNfsf8+Dv/WWv8f50wkLO+1odT7/M7yft0zmNTcfyzWQf26EjPznkM7duNrWU7eemjtXHL/eSEgfz3Nw/hPx56l/dWFtWafvKQXryZX1hvm8HHpbi8kpunL9rtzGK0S8b0363L46VjBvDo3JUNmn+0t687cbfumamgRL6ZKZEXEWmd8tdv483PNnDW1/pyQI/Gb6zLd1YxM7+QI/p1o3NeLtvLK+ucT/nOKj76YgtH9d+r3q5nc5ZuZFlhMecd1Y8u7duypXQHv31jKd06tOXK8YOoco4XP1xLl/a5TBi2X9ydnqpqx3srizi8b1e6tq99TYxzjtIdVXTKy+U3ry3m5QXrueG0Qzl92H61yjbVKwvX8+g7K7ng2AM4e0RfZi0upKKymm8evm/NztZXJTvo1qEtOTnG6qJS1m4p45j+e1NUuoObpy2iQ9s23Hb2UDq2y6V8ZxV//3gd+3ZrH3dne9biDbzwwVomHLEf1zzzMRWV1fzyrMO5cFT/RrX71Pvm7Ha0PjfH+P7xA7jh1EMxM95aupFlG7Zz3tH707V9W6qrHTe9tJD3CoqoqnYM7dON288ZRrcOteNfsKmE70+dT/u2bXji0mPo0TmPZ+Z/wS3TP+X4QT15+MKRzMwv5NZ/fMoJg3vxq4lDa+puLduJc46u7dty8r1vsWJTCUf068b0y8dgZjz/wRoWrtnCGcP7cPO0ReTl5vBV6U6+KNq1Y1IwZQIbtpXzyeotjDukJ4btthO7aO1WNhVXcPygnuysqmbu8k306d6BBWu2MH5Ib/YJBw4ipn20lquf2bUTc8c5w7jxxYXs3akdb193Ys3O94Zt5fTqksfqojJunr6ILu1zufvc4Rx682s1da85eTCXfL0/d7++hCf+tYp9u7bnjm8P4+sH71Oz7hRuK2f0lFm1zg4dtl9Xpl0+hj/M/pzt5Tu59pTBdGzn33vFxmKmziugqHQn7XNzOP2I/RjQoxMnRN31LeJbw/tw//kjEi8cLUSJfDNTIi8iIpK9CreVs3ZLGV+r4yxPIs45Vm0uxQwO7FG7a1hTRfKm2Acd5uU27Ewe+B2g+QVFHD+oZ9wzgJEzbeu3ljHurtnsqKzmD/95JKc18w5bVbVjwv1vs3TDdm6bOIwLjj2AlZtK6N01ryaRrsvLC9Zzw4sLOO6gHjw0aSRmhnOOT9dtY1DvznFj8tm6bSzfWExebg4/fvIDzOD1qxvfHeaBmcu4542l5Bjk/+pUnKPW0+pTRYl8M1MiLyIiIq1BUckOtpTu4KCenesvnITqasdXpTtquvk1VmVVNblJ3ihheWExHdq1oW+4Filb6YFQIiIiIlLL3p3asXenxLcxbqqcHEs6iQeSTuKBWtdlSOO1/L3mRERERESk2SmRFxERERHJQkrkRURERESykBJ5EREREZEspEReRERERCQLKZEXEREREclCSuRFRERERLKQEnkRERERkSyU1kTezPqZ2aNmts7MKsyswMzuM7O90tkuEREREZFMl7Ynu5rZQGAe0AuYDiwGjgGuAk41szHOuc3pap+IiIiISCZL5xH5/8Un8Vc65yY6525wzp0E/BY4BPh1GtsmIiIiIpLR0pLIm9lBwClAAfBgzORfACXAJDPrlOKmiYiIiIhkhXQdkT8pDGc456qjJzjntgNzgY7AcalumIiIiIhINkhXIn9IGC5NMH1ZGA5OQVtERERERLJOui527RaGWxNMj4zvXtdMzOyDBJMOTaZRIiIiIiLZIlPvI29h6NLaChERERGRDJWuI/KRI+7dEkzvGlMuLufcyHjjzWxzfn5+x5Ej404WEREREWkW+fn5AP3T8d7pSuSXhGGiPvCDwjBRH/r6bCsrK+PDDz8sSLJ+siJdehan+H2zneKWHMUtOYpbchS35ChuyVHckqO4JaepcesPbGuepjSOOZf63ivhYVDL8befHBh95xoz6wKsx3f76emcK0l5A5MU6bOf6EyBxKe4JUdxS47ilhzFLTmKW3IUt+QobsnJ5rilpY+8c+5zYAZ+D+bymMm3Ap2AJ7IpiRcRERERSaV0da0B+AkwD7jfzMYD+cCxwIn4LjU/T2PbREREREQyWtruWhOOyh8FPI5P4H8GDATuB0Y55zanq20iIiIiIpkunUfkcc6tBi5JZxtERERERLJRpt5HXkRERERE6pCWu9aIiIiIiEjT6Ii8iIiIiEgWUiIvIiIiIpKFlMiLiIiIiGQhJfIiIiIiIllIibyIiIiISBZSIi8iIiIikoWUyIuIiIiIZCEl8s3AzPqZ2aNmts7MKsyswMzuM7O90t225mJmPczsB2b2kpktN7MyM9tqZu+Y2ffNLO6yZGajzewVMysys1IzW2BmV5tZmzre6wwzmx3mX2xm/zazi+pp30Vm9l4ovzXUP6Opn7ulmNkkM3Ph9YMEZVo8DmbWJnwfC8J3WhS+r9FN/YzNxcyON7MXzGx9WL/Wm9kMMzs9Tlktb4CZTQgxWhO+1xVm9pyZjUpQfo+Im5l9x8weMLO3zWxbWP+eqqdORsYmletuY+JmZoPM7Hozm2Vmq81sh5ltMLPpZnZiPe/T4jEwsw5mdquZLTGzcjMrNLNnzWxIwyPSMMksbzH1H7Fd24mDE5RJSQzMbG/zeU2B+d/hdebznn4N/TwNleR6amH5mR1iUGZmK8PnGpygTutY3pxzejXhBQwENgAOmAZMAWaF/xcDPdLdxmb6nJeFz7QOeBq4A3gU2BLGP094wFhUnbOASqAYeAS4K8TEAc8leJ+fhumbgAeB3wKrw7i7E9S5O0xfHco/CGwO436a7tjFae/+IW7bQxt/kI44AAY8F7Ws3hW+p+LwvZ2VAbH6n9C+jcBjwO3AQ8B84E4tb3Hb95uoz/Tn8Jv0PLADqAa+t6fGDfg4vN92ID/8/VQd5TMyNqledxsTN+BvYfqnwJ/w24oXQ7sccGW6YgDkAe+EOvPDuvIXYCdQAhybzuUtpu6ZUXUdcHC6YgD0AJaEOjPxvynTwv8bgIPSvJ62B/4vKg6/D8vdVGAFcEZrXt6aLfB76gt4PXxJV8SMvzeM/2O629hMn/Ok8MOSEzN+X+CL8Fm/HTW+K1AIVABHRY1vD8wL5b8bM6/+QHlYmfpHjd8LWB7qjIqpMzqMXw7sFTOvzWF+/Zvy2Zs5jga8CXwefghqJfKpigNwfqgzF2gfNf7o8L0VAl3SGKtzQ/veiNcOoK2Wt1ox2ReoAr4EesVMOzG0fcWeGrcQg0FhPTyBuhPSjI0NKV53Gxm3i4ERccaPw+9MVgD7pSMGwI2hznNEbcvwO2yRnY+c+uLREnGLqdcTvw7/DZhN4kQ+JTHA75A54N6Y8VeG8a+laz0N5R8MZW6P9/0Rta1ojctbswV+T3wBB4UvY2WcBb8Lfk+tBOiU7ra2cBxuCnF4IGrcpWHc1DjlTwrT3ooZ/8sw/tY4deLOD3gijL8kTp2E80tjrK7CHxUdC0wmfiKfkjgAc8L4E+PUSTi/FMUpB38kpQTo2YDyWt58G44NbZieYPo2YLvi5qD+hDRjY5POdbe+uNVTdwYxB31SFQN8UrgqjB8Qp07C+aU6bsBL+ES+B3Un8i0eA6ATUIrPZ2IT1Rx8/uNo5qPyDY0bvldEFfAeMb0C6phnq1re1Ee+aU4KwxnOueroCc657fg9t47AcaluWIrtDMPKqHGR2LwWp/wc/A/DaDPLa2CdV2PKNKVOWoQ+cVOA3znn5tRRtMXjEOI+Gv89vN2I90mV0cAA4BXgK/N9vq83s6ssfj9vLW/eMvxRz2PMbJ/oCWY2Fn+A4c2o0YpbYhkZmyxYd+sSb1sBqYnBQOAAYKlzbmUD66ScmV0MTAQuc85trqNcqmIwCugAzA15TY2Q98wI/9Z5/UMLOh+/QzEV6Gpm3zOzG83sR4muK6CVLW9K5JvmkDBcmmD6sjCMe6FFa2BmucCF4d/olSJhbJxzlfi9+Fz8WY2G1FmPPzrbz8w6hvfuBPQFisP0WBkT/xCnJ/HdkG6qp3gq4nAw0AbfzSJ2o5qoTiodHYYbgA+Bf+B3gu4D5pnZW2bWM6q8ljfAOVcEXA/0Bj4zs4fM7A4zexa/wX0D+HFUFcUtsUyNTaavu3GZ2YHAeHwyNCdqfKpikPHb6xCj3+GPPk+rp3iqYpDpcYtsK7rhu6w+ie9i8ydgqZk9aFEXprfG5U2JfNN0C8OtCaZHxndPQVvSZQowFHjFOfd61PhkYtPQOt1ihtkQ/1uAEcDFzrmyesqmIg6ZHrteYXgZ/mjQyfijyUPx16WMxfc7jNDyFjjn7gPOwSeZPwRuwF9vsBp43DlXGFVccUssU2OTdfEMRzSfxl/8N9k591XU5FTFIKPjZv7Ob1PxXViubEAVxc2LbCt+CbwPDMNvK8bjE/ufADdHlW91cVMi37IsDF1aW9FCzOxK4Gf4K7gnNbZ6GDYmNsnGM63xN7Nj8Efh73HO/as5ZhmGLRmHdC+7kSMoBnzHOTfTOVfsnPsUOBtYA4xL0M0mnj1pebsOf5eax/GndzsBI/HXHDxtZnc2ZnZh2OrjloRMjU26193dhKOhTwJjgGfwdwtJRkvHIN1xuwZ/QfAPY3Z0kpWqGKQ7bpFtxXrgbOfcorCtmAV8B39N2rVm1q6R882auCmRb5rYoyuxusaUazXM7HL8KcDP8BdrFMUUSSY2Da2zrYHl69sjbnFRXWqWsvtRgbqkIg6ZvuxGNmQrnHOfRE8IZzQiZ3+OCUMtb4CZnYC/xdnfnXPXOudWOOdKnXMf4neA1gI/M7NIdxDFLbFMjU2mr7s1QhL/FP6M0LP4W5/GJi6pikHGxs3MBgG/Bh5zzr3SwGqpikHGxi2IbCteiz3bHbYdK/FH6CP3bW91y5sS+aZZEoaJ+jgNCsNEfaSykpldjb9P6yJ8Ev9lnGIJYxOS2wH4C55WNLDOfvgji2ucc6UAzrkSfGLSOUyPlQnx74z/PEOA8qiHezjgF6HMw2HcfeH/VMRhOf5K/4PC99GQOqkUicGWBNMjP94dYsrv6ctb5GEm/4ydED7He/jf/RFhtOKWWKbGJtPXXaAmRn8Fvou/d/YF8foXpzAGmby9Phzf7eiS6G1E2E6MC2WWhXETw/+pikEmxw0aua1ojcubEvmmiWwsT7GYJ5uaWRf8qcQy4N1UN6ylmNn1+IcnfIxP4gsTFJ0VhqfGmTYWfzefec65igbWOS2mTFPqpFIF/qER8V4fhTLvhP8j3W5aPA4h7vPw38PxjXifVJmDT5IGJTglOjQMC8JQy5sXuYNKzwTTI+N3hKHillhGxiYL1l3COvs8/kj8E8Ak51xVHVVSEYPP8TcbGGxmAxpYJ1UKSLydiBwoey78XwApjcG7+DxmTMhraoS855Twb62DBykyMwyHxk4I12ZEEuaCqEmta3lr6v0r9/QXe8gDocJnujl8pveBvesp2xX/NM7GPExlAFn6oJkk4zmZ+PeRT0kcaNgDLrqmMT5PhfbdFjP+G/h+j1uA7lredmvfeaF9XwJ9Y6adFuJWRnji9J4cNxr2QKiMjE06190GxC0PeDmU+TMNeOBNqmJAih8I1Zi41VFvNonvI5+SGLDrgVD3xIxvkQdCNXJ5a4dPmquBb8RMuy3Und2al7cWCfye9MJfTLYhfCnT8I8FnhX+X0LYYGb7C7gofKZK/BH5yXFeF8fUmciux5v/GbiTqMebE+fhDcAVYXpjHm9+T5ge/ajlTWFcSh79nmRMJxMnkU9VHNj9kdP54ftpsce8JxGfXvhbdDn8Efq7Q3sr8fejPlfLW6225eBvMenw/bCnEvrM4zd0DrhqT41b+KyPh9dr4b0/jxp3d5zyGRcbUrzuNiZuwGNh+kbgVuJvK05IRwzwOxlzQ535+Luu/QX/e1ICHJvO5S3BPGaTOJFPSQzwD6ZaEurMxOc508L/G4CBaV5Pv46/rWlliMfdwFuhXiEwuDUvb80W+D35BeyP//Fajz9lvQp/IWidR62z6cWupLOu1+w49cYQHuqDPxK4EH91fps63uvMsBJuDwv7fOCietp3UShXEuq9BZyR7rg1MKa1EvlUxQF/i8JrwvdSFr6nV4DR6Y5PaN/e+LNbK8O6tRmYDhyXoPwev7wBbYGr8afEt4WNTCH+Xvyn7Mlxa8DvWEG2xCaV625j4sauxLOu1+R0xQDfV/pW/EGCCvwOx3PAYZmwvMWZRySetRL5VMYA/1v8O3x+swOf7zwK9MuEuAGH4e+KVBjatxp/JiFh+1rL8mbhjUREREREJIvoYlcRERERkSykRF5EREREJAspkRcRERERyUJK5EVEREREspASeRERERGRLKREXkREREQkCymRFxERERHJQkrkRURERESykBJ5EREREZEspEReRERERCQLKZEXEREREclCSuRFRERERLKQEnkRERERkSykRF5EREREJAspkRcRERERyUJK5EVEREREspASeRERERGRLPT/gWKHwWt6eXYAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2adfe3a76d8>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 377
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['train'], label='Training loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 显示测试Loss\n",
    "迭代次数再增加一些，下降的趋势会明显一些"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAvIAAAH0CAYAAABfKsnMAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzs3Xd4FVX6B/DvJPTeRBALoKJYEEGxoLCI7q6KiG131RXXsru6a91lbWthLYgi/lSKIkWkWClSRUAIJfSEkhBKSEgnkN7rvfP7IySk3DLlTL3fz/PwkNw798x7507uvHPmnXMkWZZBRERERETOEmZ1AEREREREpB4TeSIiIiIiB2IiT0RERETkQEzkiYiIiIgciIk8EREREZEDMZEnIiIiInIgJvJERERERA7ERJ6IiIiIyIGYyBMRERERORATeSIiIiIiB2IiT0RERETkQEzkiYiIiIgciIk8EREREZEDMZEnIiIiInIgJvJERERERA7ERJ6IiIiIyIGaWR2AmSRJOg6gA4Aki0MhIiIiInfrDaBQluU+Rq0gpBJ5AB1at27dpX///l2sDoSIiIiI3OvQoUMoKyszdB2hlsgn9e/fv0tUVJTVcRARERGRiw0ePBjR0dFJRq6DNfJERERERA7ERJ6IiIiIyIGYyBMRERERORATeSIiIiIiB2IiT0RERETkQEzkiYiIiIgciIk8EREREZEDhdo48kREROQCXq8Xubm5KCoqQkVFBWRZtjokcjFJktCyZUu0b98eXbp0QViYPfrCmcgTERGRo3i9XqSmpqK0tNTqUChEyLKM8vJylJeXo6SkBOedd54tknkm8kREROQoubm5KC0tRbNmzdCjRw+0bdvWFkkVuZfX60VJSQkyMzNRWlqK3NxcdOvWzeqwWCNPREREzlJUVAQA6NGjB9q3b88kngwXFhaG9u3bo0ePHgDO7INW455PREREjlJRUQEAaNu2rcWRUKip3edq90GrMZEnIiIiR6m9sZU98WQ2SZIAwDY3V/MvgIiIiIhIgdpE3i6YyBMRERERORATeXIlu1zyIiIiIjIKE3lylcLyKtw7PRK3frwJx04VWx0OERERBXHNNdegXbt2VofhSEzkyVU+XHMY0Sn5SMgqwdMLoqwOh4iIyBCSJKn6N3fuXEPjKS4uhiRJGDVqlKHroYY4IRS5yvaEnLqf49kjT0RELvXWW281eeyTTz5BQUEBnn/+eXTq1KnBcwMHDjQrNDIRE3kiIiIihxk/fnyTx+bOnYuCggK88MIL6N27t+kxkflYWkNEREQUQrKysjBu3DhccsklaNWqFTp37ozf/e53iIiIaLJsWVkZPvroIwwcOBCdOnVC27Zt0adPH9x7773YvHkzAGDq1Klo3749AGDVqlUNSno++ugjzXF6PB589tlnGDRoENq2bYt27drh+uuvx5w5c3wu/+uvv+L2229Hr1690LJlS/Ts2RNDhw7FBx980GC5jIwMPP/88+jXrx/atGmDzp07o3///njiiSeQmpqqOV4rsEeeiIiIKEQcPXoUt9xyC9LT0zFixAjceeedKCwsxPLlyzFy5EjMnz8fDz30UN3yf/zjH7FixQpcffXV+Mtf/oKWLVsiPT0dmzdvxoYNGzBs2DAMGTIEr776Kt5//31cfPHFDV5/4403aorT6/Xivvvuw7Jly9CnTx/8/e9/h8fjwZIlS/DEE09gx44d+PLLL+uWX7x4Me6//3507doVo0ePRo8ePZCdnY24uDjMmDEDL7/8MgCgsLAQ1113HTIyMvDb3/4WY8aMQVVVFZKTk7Fo0SI88sgjOO+88zRuXfMxkSdX4aCTRERE/j388MPIzMzEsmXLMHr06LrHc3JyMHToUDz11FO444470KlTJ5w4cQIrVqzAsGHDEBER0WAyJFmWkZubCwAYMmQILrvsMrz//vvo16+fz7IftWbPno1ly5bhxhtvxPr169G6dWsAwDvvvIMbb7wRM2fOxKhRo+reQ21Sv2PHDlx00UUN2srOzq77edWqVUhLS8Prr7+Od955p8Fy5eXlqK6u1h27mZjIExERkav0fmWV1SEoljTxTtPWFRkZiT179uAvf/lLgyQeALp27Yo33ngDf/7zn7F8+XKMHTu27rmWLVs2mdFUkiR07drVsFhry2cmTZpUl8QDQIcOHfDee+9hzJgxmDVrVoP3IUkSWrVq1aStbt26NXmsfpu1fL3W7pjIk6vYa+JkIiIi+9i+fTuAmhp5X73m6enpAIBDhw4BAHr27IkRI0Zg3bp1uOaaa3DPPffg5ptvxpAhQwxPevfu3YtWrVrhhhtuaPLcLbfcUrdMrYcffhhr167FwIED8cc//hEjRozA0KFD0bNnzwavve2223DWWWfhjTfewLZt23D77bdj6NChGDBgAMLCnHfrKBN5IiIiohCQk1MzRPOqVauwapX/qxbFxWeGb16+fDkmTJiA77//Hq+//joAoE2bNvjTn/6ESZMmoUuXLsLjLC8vR0VFBXr37t3kSgAAtG/fHm3btkV+fn7dY2PHjkW7du3wySefYMaMGZg+fToA4Prrr8fEiRMxfPhwADW98zt37sT48eOxcuXKuu1w9tln47nnnsPLL7+M8PBw4e/JKEzkyVVYI09ERGaWqzhJx44dAdTUnz/++OOKXtOuXTtMmDABEyZMQHJyMjZt2oTZs2djzpw5yMjIwM8//yw8zlatWqFly5Y4efKkz+eLi4tRUlKCXr16NXj83nvvxb333ouioiLs2LEDy5cvx4wZM3DHHXcgJiYGffv2BQD06dMHX3/9NbxeL2JjY/Hrr79i6tSp+O9//4vw8PC6G2OdwHnXEIiIiIhIteuvvx4AsGXLFk2vv+CCCzB27Fj8+uuv6NWrF9auXYuysjIAqOvF9ng8QmIdOHAgysrKsHPnzibPbdiwAQAwaNAgn69t3749brvtNkyZMgUvvvgiSktLsW7duibLhYWFYcCAAXjxxRexcuVKAMBPP/0kJH6zMJEnIiIiCgHDhw/HoEGDsGDBAnz77bc+l9m7dy/y8vIA1Iy3Hh0d3WSZoqIilJSUoEWLFnUJfOvWrdG6dWukpKQIibX2isFLL72EioqKBuuuLfF54okn6h5ft25dg+Vq1fbqt2nTBgCwb98+pKWlBV3OKVhaQ0RERBQCJEnCjz/+iJEjR+Khhx7C5MmTce2116JDhw5ITU3F3r17cfjwYcTExKBz585ITEzEzTffjCuvvBIDBw5Er169kJ+fjxUrViA/Px+vvfYaWrRoUdf+yJEjsXLlStx333248sor0axZM9x66611VwLUePLJJ7FixQqsXLkSV1xxBUaPHl03jnxqaioef/xx3H333XXLP/3008jLy8Pw4cPRu3dvhIeHY+fOndiyZQv69euHe+65BwCwcuVKvPXWW7jppptwySWXoFu3bkhOTsayZcsQHh6OcePG6d/QJmIiT67CUWuIiIj869u3L/bu3YtPP/0US5cuxbx58yDLMnr27InLL78c//nPf+rGYb/00kvx5ptvIiIiAuvXr0dOTg66du2K/v3745NPPsH999/foO0vvvgCL7zwAiIiIvDTTz/B6/WiVatWmhL5sLAwLF26FFOnTsXXX3+Nzz//HJIk4fLLL8ebb77ZoDceAN566y2sWLEC0dHRWLt2LcLDw3H++edj/PjxePbZZ9GuXTsAwOjRo5GVlYUtW7ZgyZIlKC4uRs+ePXHXXXfh3//+N6655hqNW9YakiyHzu2BkiRFDRo0aFBUVJTVoZBBbpkcgcSskrrfecMTEZH71A6P2L9/f4sjoVCkdP8bPHgwoqOjo2VZHmxULKyRJyIiIiJyICbyRBqtOnAC/1txEGl5pVaHQkRERCGINfJEGhzPLsE/v6m5k39vSj5++udQiyMiIiKiUMMeeSIN1sedmaRiX2p+gCWJiIiIjMFEnoiIiIjIgZjIE2kgI3RGeyIiIiJ7YiJPRERERKSA3YZtZyJPREREjiJJNdP/eb1eiyOhUFObyNfug1ZjIk/uYq8TZSIiMkDLli0BACUlJUGWJBKrdp+r3QetxkSeiIiIHKV9+/YAgMzMTBQVFcHr9dqu5IHcQ5ZleL1eFBUVITMzE8CZfdBqHEeeiIiIHKVLly4oKSlBaWkp0tLSrA6HQkybNm3QpUsXq8MAwESe3MYeJWtERGSgsLAwnHfeecjNzUVRUREqKirYI0+GkiQJLVu2RPv27dGlSxeEhdmjqIWJPBERETlOWFgYunXrhm7dulkdCpFl7HE6QSQKO2SIiIgoRDCRJyIiIiJyICbyRBqwFJOIiIisxkSeiIiIiMiBmMiHiH2p+Xjmm2isPJBhdSjG4qg1REREFCI4ak2IGDMtEgCw8sAJDL2wGzq3beF32ZKKahzMKMTgCzojPMxhmTFLXoiIiChECOmRlyTpfkmSpkiStEWSpEJJkmRJkhYIaPeR023JkiQ9KSJWApJy/E9p7fXKuGvqVvxhxna8uSxWyPoKy6twPJvTaBMRERGJJKq05nUAzwAYCCBdRIOSJJ0HYAqAYhHtkTJ7kvOQmFWTdC/cmQIASMouwQNfbMPTC6JQXuVR1V5+aSWGvr8BIz6KwNK9nH2PiIiISBRRifyLAPoB6ADgab2NSZIkAfgKQA6AL/S2R8pVVnubPPbMt9HYnZSHn2Mz8eXmRFXtTV57FEUV1QCAF7/fLyRGO2AFDxEREVlNSCIvy/JGWZbjZXHzIz8H4BYAjwFgTYbFYtML637eeOSUqtfmllaKDoeIiIiIYMNRayRJ6g9gIoBPZVnebHU81JDqW19d2nXtsFuAiYiIyIVsNWqNJEnNAMwHkALgNR3tRPl56lKtbYYK2a2Zt2DcSkRERGQ1WyXyAN4EcDWAm2RZLrM6GBKAXddEREREhrBNIi9J0hDU9MJPlmV5u562ZFke7GcdUQAG6WmbVGLXNREREZEhbFEjX6+k5iiANywOhwKoGVDIvnjeQERERKHCFok8gHaoGb6yP4DyepNAyQDeOr3MzNOPfWJZlCFA2LhDRERERGQou5TWVACY7ee5Qaipm98K4AgAXWU3FFiwPN5O/fHf7UrBlmPZeGbERejfswMAe8VHREREZCTTE3lJkpoDuBBAlSzLCQBw+sbWJ/0sPx41ifzXsizPMitOsrfErGK8siQGABB5LBv73vytqevnlQsiIiKympBEXpKkMQDGnP61x+n/b5Akae7pn7NlWR53+udeAA4BSAbQW8T6qalDJwpx6EQhbr+iJ1q3CLc6HOH2JOfV/ZxfWmVhJERERETWENUjPxDAo40e63v6H1CTtI8DmSK3pBJ3T4tEZbUXR04W4dXb+1sdkmnYUU5EREShQsjNrrIsj5dlWQrwr3e9ZZMaP6awbZbVKDQ38jgqq70AgBmbEoW2bfNBa4iIiIhChl1GrSGB9PRKyyz+JiIiInIEJvKkimTzcWGsii4uo9CiNRMREVGoYiLvANMjjuHhWTuwNyUv+MIhTu/1hGOnilFW6VGwnoZreuCLbTrXTERERKQOE3mbO5hRgA/XHEHksRzcM934ZFF0YU3jhNfOZm1JxK0fb8KIjyJQUR08ma+vREHyT0RERCQSE3mbO5BWYHUIIePdVYcAAJmF5fhhd2rAZe1eYkRERETux0Te5ky/91Tw1K5OTXgLy6sDPu+kKw1ERETkTkzkSRW1ablTEt780kqrQyAiIiJShYm8C+npxV+4M1lcIA4y8efDVodAREREpAoTeZszu0d7/aFTQttzSmlN/KniBr+H4sRX2xNycM/0SEzdEG91KERERKRAM6sDIHdzSmkNAQ/O3AEA2JuSj99d3gMXn93e4oiIiIgoEPbIkyqh2FMdio6eLA6+EBEREVmKPfJkufIqDyasPoSSCg8ev6k3zunYGp3btrA6rIBMH02IiIiIqBEm8qSKETXvs7YkYt72mptsF0enoUWzMKx9YRh6d2uruI3pEcfw9PALhcdGREREZFcsrbG5UOj5XbgzpcHvldVevLT4gKo2PlxzBKtiTogMi4iIiMjWmMiT5XydrJwqLFfdzjeNTgj0cMpoO0RERBS6mMiT5XyNbBP0QoSPBULh6gURERFRLSbyNme33HR7Yg7G/bgfsk2zZrvGRURERCQaE3lSbVFUGn45mGltED4qX7SOWb8u7iSikvN0BuQuHP+fiIjI/pjIkyYx6QVWhyBEYXkV/jpvj9VhEBEREanGRN5h8koqVb/mREGZ8DhE3gyqqRrGz2sklTNWJWeXBnz+/Z8P4YEvtmF/ar6qdomIiIiMxkTeYd5ZFaf6NUMnbhAeh9J82cyS9R2JuTieXSKsvW3HsjFjUyJ2J+Xh/i+2CWuXiIiISAQm8nbXKBNeEp2uugmvG8udDR4dUpKA6JQzdfNVHjduRCIiInIyJvKkicg82leKbIfBZ+wQAxEREZE/TORdKCRGHBH0FkNiWxEREZErMZEnbVTeVGqz5oOqqPIyxSciIiJbYyLvQO+vPoSKao+lMQgtrdEyS6vBif7/rT+Kj9cdNXYlRERERDowkbc5X/nsjM2JmBuZZHYoRERERGQjTOQdatbW45au3+rSF1F1L1rHw5d5JywRERFZjIk82YB1STFvdiUiIiKnYiJPmoic2VVjANau3scliWOnii2IhIiIiEIVE3nSRGRpja8qleziClsnxp9HJDR57NaPN2FupLUlT0RERBQ6mMjbXKiWYpdWenDrx5uwJvaE7wUs3C4JWcUorqj2+dz4FXEmR0NEREShiok8BeTxWnsm8dSCaEvX70v8SfteKSAiIqLQ0czqAOiMymovErKKcWmP9j5rsOvT01MfrO36xkyL1L4iBI5z1/Fc5JZUolrLyYLVo+YQERERWYyJvE3Isox7pkfiYEYhHh/aB2+M6g9JkgwZ5lBNmzHpBT4f15tHx6YX4A8ztutsRb9QLV0iIiIi52NpjU3sTyvAwYxCAMCcyOMYOXkT4k8W+V3e6nHc668/v7QSH689gh/2pAZcrr5XlhzQFwATcCIiIgpx7JG3ifIqT4PfE7NL8Lf5UXj0hgssiki5CasP4Yc9aQCAczu3xo0Xdqt7zl+Pt9drRmSBebwyXlkSo/p1Vp9EmYFXKoiIiOyPPfI2djy7xO9z9RMtWZZx7FSx4htT1dTIK2mjNokHgLmRSQpfrzcAna8H8O2uFBw6Uai/ISIiIiILMJF3gdeWxuLWjzfh0Tm7FC1vRN29WmoS+Ydm7kBldaMufAFvYcPhU/obcalQuOoQqjYfzcK/f9iP6JQ8q0MhIiKdmMjbnJJ89dtdKQCArceykVlQ7rqyiG0JOfjb/D1Wh1EnFHJcu+9DeSWVeP/nQ5i3PckWJ6ZOUV7lwdg5u7A4Og33Tt9mdThERKQTa+RtQm1y6K/HtMpjTvG52T22EUeyUFRehfatmpu7YrKld1bFYUl0OgDgvM5tMOLS7hZH5Ay5JZVWh0BERAKxR96h/HVCVnm8mB6RYG4w9SjtG5U09GvHn6o3EVModIuTX7VJPADM3ZZkXSBEREQWYiJvA6cKy7E3NV9IWwt3pghpJxgtiXiD1+tNxFlNQURERCGOpTUWyy2pxLBJG1Fe5bskRm357+ajWQKi0s7IjnJ2wpMVqj1eeGQZLZuFWx0KERFRA+yRt9iUDfF+k3gtvCbd+OevR73x2mV2nZODnSwsx7APN+LG9zfgcCaHKiUiInthIm+x0gpP8IVUUDJGfFRyHuZGHkdBWZX29Wh+pfbXNzglENA9r3W0ExHj8JMz/HdpDDIKypFTUom/zrPPyElEREQAS2ssJzonPFb/hlA/3l11CAAwfkUcDr/ze7RqblzJgN9aeg1v/GRB+Zlf2NFPJohJL6j7OTW3zMJIiIiImmKPvAV+2JOK3q+swpNf7w66bGy9RMIIMzcnanqd0jzcX2mNlvOXcT/u1/Aq/7T2rLM/noiIiOyAibzJqj1evLToAABg/aFTQW9OXbI3PeDzek1ed9TQ9kUqqRRbhlRcXi20vVCWkV+GB77Yhkdm70RhufaSLV9SckoNP6H1x21zTbEqjIjIXZjIm+xkUUWD3zPql4s4iNLhJ/UOUxmgYd0qTJo8KxSM+3E/diflYUt8Nj5cc1hYuwlZxRj+0UaMmrIVa2JP+FyGyalybjsxISIKdUzkTbYkKs3qEITQXVrDceRdZVtCTt3P6+JOCmv35UUH6pLPpxZE+1yGySkREYUqJvImc1Ipi5HYiUpKFLH8SShevSAichcm8g5WWlmNt1fEWR2Gc2keflJwHEQm4dWL0LEkOg3vrYrDqUJnlm8SkTIcftIkReVV2F6v/ECEzyMSMCfyuLD2qgXUjIt+j6LM2XocsRkFeGFkP5zftY3V4ZBAPLEiaig2vQD/+qFmlK/4U8WY+9gQiyMiIqMwkTfJX77ajajkPKFtTo9IENZWen4Z7v98m+LlJUlCWaUHy/c3HFWnuKIaheVV6NCqedDXm2VPUi7eXllz5eLoySKsfPZm09ZNZCc86QkNS6LPfC9HHAk8MhoRORtLa0xQ5fEKT+IBsXXmLy86gBMqRtCRAEyPOIaXF8c0ee5QRvCp7M3MJ345mFn3c2x68NhI373Eho1U5AfLRYiIKFQxkTeB16BMQ2Tv2r7UfNWvmbLhmM/Hw8KCB2Zmz2B4mO/dXO2n8s3OFFR7vOzVDCE8RyAiIjtjaY0JjOoxrClPsSbVCJTMhgV4cuHOZJwsKEdhmXmjkYQLOl19bWkMWjUPQ6c2gcuGnMLrlf2edDnpXIUnVkREFKrYI+9govKXoyeLhK47UIf8f5fG4rMNx3BEwzrrUxNzoBMLtd74KVZYW1aatvEYBvxvLSavPeLzefZE1+A5AhER2RkTeROk5pYa0q6o/HT01K0orhDXQy4ycfalvMqDWVuVj9bjL55Qrq2e9MsRFFdUY8qGY6io9lgdjm2F8C5CREQOwETeBM9+u9eQdkXdVFhepX7YyUCjzhidyEenqLtxOFxBzb5SZo62Y5bDJ/RdHWnMhZuIiIjIlpjIm+BwpthEqZbA/FSo+omcEb3eHq+6RkVvJ7NHZTHak/P2CG0vlK90EBERmYmJvGPJlvYOa73ZVYRqFYl8WaXH7w2dsobCCZElSHaRVVRhdQhEJBCvihGFDibyDpVdXGlpUhnwZleD9yqPR3kCft2E9TiRL3aK8h/2pAptz22YRBBZi1fFiEIHE3kSLtzgTM6j4ihVWF6N+TuSfT6n9WD3c2xm8IVICKtPCtyWELmtLIyIKNQxkSfh3l4Zh+PZJYa1r7ZGvrFZWxIFRUJGc1sibTUt5WTkPFafABOReZjIk3Bb4rPxhxnbsTrmBArKqoS3r6ZG3pd3Vx0SFAm5HRMiIrIrj1fG8v0ZWL4/Q3cHFzkXZ3YlbYJkOFlFFfjHwmjhqz2eXQKvoC8s9vZSMG7bR1haQ+Qeq2NO4LnTw1uHScCoAedYHBFZgT3ypIlV6cDz3+3V3SNfK+5EoZB2qCGR+wZ7xImIfKs/R80z3xgzXw3ZHxN5cpQDaQXweNVPYGV3FdUevLokBv9cGI1TRWJH2XEyJT3izPWJiChUsbSGNLGyp7So3H1juc/eehzf7koBAFR5vPhy7DWa2skprsCe5DwM73cWWjUPFxmibbms+oWIiEgxJvLkOCJuVr3v820CIhFncVRa3c9r405qasPjlTFmeiRSc8sw+qpz8NmDV4sKzzIsrSEiIvKPpTWkidNvmotKzrM6hAZE9CrvT8tHam4ZAGD5/gwBLRL7+4mIyM6YyJMm7Cm1H1Gj+TgNd0Wihvg3QRQ6mMgTUROy28Zd1IwpERER2RcTedIks4Ajq5BvksmXa4w95eAJTX3HThXh2W/3Yv72JKtDISIi8GZX0mjaxmNWh+AuNssX9STjInvzeWHAXh6dsxvp+WVYsT8Dgy7ojMvP6Wh1SEREIY098qQJ8yv/KqudP869k0prWPxinvT8srqftx3LEdZuRbUHi6LSEHksW1ibREShQEgiL0nS/ZIkTZEkaYskSYWSJMmSJC1Q2UZXSZKelCRpqSRJxyRJKpMkqUCSpK2SJD0hSRJPOmwknHe7+jVra6LVIQRkdJIusrTGbrtZam6p1SHoYrftWevrbUkY9+N+PDxrJ+IyOOMyEZFSopLj1wE8A2AggHSNbTwAYCaA6wDsBPAJgMUArgAwC8APktnFt+RXGE+r/NqeoL6n0m79327+U/N4Zc0nM8995+xp0Bu/bT0ndSJ3kQmrD9f7Wf88EUREoUJUOvYigH4AOgB4WmMbRwGMBnCuLMsPy7L8qizLjwO4FEAqgPsA3CsiWNKPPfL65ZdWoqLaY3UYPjmptEaN2PQC3DjxV9z+6RYUllepfv3elHwDonIml+4iRESOIiSRl2V5oyzL8bKOo78syxtkWV4hy7K30eOZAL44/etvdIRJAoWFMZHXY/PRLAyZ8CuGTtyIvJJKq8MJGY/O2YWThRU4nFmESWuOWB0OERGRLk4Ztaa266za0iioTjgTeb+UnM6OnbMLAJBdXIGJPx82tQc8lHtSc+qdNB1IL7AwEmvIjYq4ZFl7iQwvyhERWc/2ibwkSc0AjD396xqFr4ny89SlQoIihPEoLkxqnrNvoHQsBWc0oXzSQ87Fr2ei0OGEWxYnouaG19WyLP9idTBUw6011E51JLMImYWcpItIi1OF5fjv0hh8uTnBFd9tLngLRKSQrXvkJUl6DsC/ARwG8IjS18myPNhPe1EABomJLrR5eaDwq3H5gtFWx5zAPxZG+3zO45WxLSEbF3dvjx4dWwEwf4Sc2PQCdGnbAud0am3ymoMIwW7LJqPWWBNGQGb//QDAy4sPYOORLADAxd3bY8Sl3U2PgYhIC9v2yEuS9E8AnwKIAzBCluVci0Oierzs8rENf0k8AEzZEI9HZu/CyMkRKNIwSoteP+1Nx6gpW3HzhxuRZrcSIu7DdFptEg8Ai6LTLIxEDL3nqPmllVi+PwP5pbwRn8jubJnIS5L0AoCpAGJRk8RnWhwSNeJll7xQRm3NT9bHAwBKKj2Ytz3ZoLX498L3+wDUXBmiY4kCAAAgAElEQVQYvzzO9PVTYHYsI5HqzdXr9cqISs5FaSXHOTDTE1/vwXPf7sXjc3dbHQoRBWG7RF6SpJcB/B+AfahJ4k9ZHBL5UFJpz/HP7UBtbmRWLlVR5Tm9PmuSN6OuCLh58irR7Je2N1W/tOat5Qdx3+fbMWrKVnYemCgqOQ8AEJ2Sz+1OZHOmJ/KSJDWXJOlSSZIu9PHcG6i5uTUKwEhZlrPNjo+IiOxh/o6aq0iJWSXYlcTqSivwPJnI3oTc7CpJ0hgAY07/2uP0/zdIkjT39M/ZsiyPO/1zLwCHACQD6F2vjUcBvA3AA2ALgOd89LQlybI8t/GDRE5m1oHSrH41f++n8frLqzz4MSoN3dq2wO+v6KG5Z92O5SFOoWfLGXUlpH5pTX1VHq/Px0ks/j0ROYuoUWsGAni00WN9T/8DapL2cQisz+n/wwG84GeZTQDmaoiPyDR2Ka1pfECu/dUuh+lZWxLx0dqjAIDv/3Y9ruvb1eKImrLLtgolVoxaQ0TkVEJKa2RZHi/LshTgX+96yyY1fkxhG5Isy78RES+R3ShJ5r/YlID7P9+G7Qk5itpcuDNFZ1SCNXqPtUk8AExcc1hzs6yRV469rRRMkyFKBe4ym49m4e5pkZgecUxco6TZoqg0vLL4AJKyS6wOhXSw9TjyRE6ktkdRSR6amFWMiT/XJLsPztyBpIl3orC8Cq8sPuD3Na//FKsqjlDDlJYjcLqVXU9ux87ZBQDYn5qPO67oid7d2locUeiKP1mEcT/uBwBEp+Rh7YvDLY6ItLLdqDVEoUZJMnX0ZHGTxz765QhWxygfmdXqkoVA69+bkm9iJET+2TMFVscJV16O57AX2EoR9eZO8HV8IedgIk/kUD/u0TZxjdHHeJt2BjbhkDB1mR5xDH+etRP7UmtOlJrO7Kp9ZwiF7ReK7H8KQET1MZEnEixYolxS0XByG0kyp7dcTQL//Hf78MPuVMPXU19segFi0gq0vViDSo+MHYk5KK9y55wIsekF+HDNEWw9lo17pkdaHQ6ZSGRpjWHfTBacMVR7vNiWkN3kO9ip9HzMVl+hJXGYyBOZ7M+zdzb43ayr4GpX89LiA0KT3Nr3+eXmBNz52ZYmz4+ashV3Td2KyGOBp484VVQuJJ5DJwrxpy934M+zduJUYTnS8kp9xOzcg93+tDPlSg5+G6SBnv3Wyft8MK8sicFDM3dizLRIV7xPF7wFEoCJPJGJCsurfNaDl5k4U66anhgtcQUaR/5kYTkmrD6MgxmFfl//xNf+p4WftSURQ977VXVMgexJzsOQCb9i2IcbsTNR2YhAbsFEgCxjQW3WoqiacsT4U8WITff/HUTkJEzkiQTzlRsVlFZh3I/78dqSGJ+vyS6uVL0etZdVtSRtouvdM/LLgi5TXuV/4p93Vx0SGU4DXhl49KtdDR7zVaJgZvmPnTnlXohQpKe0JlTO7aq9zp9gjH+DBDCRJzLFB78cxqKoNKw8cKLJc3ml6pN4QMPEUyFziNau8UmEr8vvd03dii3xWU0eJ2P5m/GVjOWGEhQiN2MiTySaj+PeNwEmZ/J4zT1Q+joue/3EIDJ5clNC8Phc/+U/dubkj4AnokTiOPm7gBpiIk/kUCIvq94yOUJcY3646bhR5XHTuyE6gwkeOV1JRTV2JuaY3klmFSbyRBbz9VWjpPda5AE3KafpiC1aBerFt+uMk6FKzz7ET5J0CY0cKyRVebw4mFFgyVVYWZZx97RI/PHLHfjvUt/3pLkNE3kiwUSUAGyObzgEo5D8V0tYDsjW7BxieZUHP8ecwMlCMUNmhjLWyJuj8fcX821S68Evd+DOz7biVT+DOxhpX2o+jp2qman2O41zoTgNE3kiG5q1JbHB7746NlSPWqMjHhGM6pwx8n3pbfv1n2Lx9MJojJ66FVUe60fJYJ15cLxqZBJuZldKzinBnuQ8ANYk0qFY9shEnsiG7FKn6sScprLai2KbzNxYO271ycIKbA0y0ZUVmNg35Yabsh3xZ+v8zWw5O37OduiwCDVM5IkEE5EHeBs1IiKhrk1QzMhTqjzeJgmRGcftrKIKDP1gA657bz32JOU2eG5NbKaqtrxeGfmlVcJiMztBdGopigvyaMvp2YTc/s7Bj4oAJvJEwjVOwq1qQ4T6YexLzcc/FkZh2b70gK9JyS3FDe9vwMjJmwyOrqm3V8Yhq6gCJZUePDRzZ93jsekFeGpBlKq2lu4N/D7tzldvu012K1tjaU1Dhu0z3MxEQjCRJxIsOiVfd++rbUbNqhfHmGmRWB2Tiee/24fs4oqAL8surkBidonu1SvpEd9w+BT+Pn8PZFlGwumbnACgst4l3hmbE329NKDvdvsf+1+P3JJKVFR7DGlbLTsm9syj9XPEJrThvuc0ej5nbn73YCJPZICo0zf7KOEr6VdyIqD2S1xk0paYpSFJl2VVMWcVVSBT4Wgvvxw8iXVxJ22fBG44fBLXTViPoRM3IF/jjL5KGV1aY1TPtR1PLojIGdxwj4taTOSJDFCk82ZLJT3yar+u5Lr/lb/S37JaviyLK6oxJ/K44uU/XHNYVfvHBVwBqM+IRPjxuXtQ5ZGRXVyJSb8cEd5+MI0/tdA75FEwoZIHsYSK3KKZ1QEQhTpfB5RgM9LJsozSSvXlGcdOFeGVxdZMkpGQVYIEFT35eSp7rJ2Wf2Tkl1kdgi0xv9JGlmVDklPDRjay+HMOxZ7bUBCKJ2hM5IkspqW05peDJzWsp6ZHOCVX+SyusgycLCxHeFjDL0czvizV3ieQkV+GgxmFxgRDZFMFZVUYO3snisqr8eXYa3BR93ZWh6QM82giIZjIE9lQsGOc2hFYatqUVSXxQE2t/98XRCHcAb0c87YnWx2C4+jplTRql2BHqToTfz6M/WkFAICnF0Rh3b+G62ovVOYWCMWeW3In1sgTWczXYTNYaY1Znpy3Bx6v3GAEGLNYfulb8HG+8dtRm0hsPpqFOz7dgo981NYXlFZh2b505AQZTcjybUrC7UzMqfs5vt6oTbZXb/f/YXcqHpq5A5uPZpm2ev4tkFuwR57IhmySx/tlxkFQ5Brc0Pc2ds4uAEDciUKMuqonLu3Roe65vy/Ygx2Jubj8nA5Y+exNPk8SvHbfqU6zuqPUDfuKHo3/tA37Uz/dbn5pJV5afAAAsC0hB0kT7zRohe4jSZLmD4jnMe7BHnkiEwRKTnw9dehEIab8Gi80Bqd9cTst3mBEJqhb47OxNyWv7oRqR2LNLLYHMwpRWO57xKQVBzKaPGbHTWz1565l9QfS8jF57REkCR45STMHnY0oHWJWNDeU1vCqQlOhuE2YyBOZINAhw9/XzuR1R1WNR+82bv861pNGvLvqEO6Zvg3f7U5V/JrPfo3nVQ4FVuzPUHX1oqLag9FTIzFlwzE8Mmdn8BfYnNv/7ojchok8kQkC9f4EmlxpT1KuEeE4glU9K7Is442fYrHruP23/atL9A0lGoKdV4qsOZipeNnUejeQp+ZySFG1uA9q54arCqKF4jZhIk9kgtD7atHO6u/hjUdOYf4OB4+Aw8RIt8lrzZ+si4j0Y2kNERnC6uQUEPsF568OWySR38cV1conz9qXki9uxRYINHxgKB3jlu/PwIxNCSjWMMtyCG2mJsxOhOzw3RiKQmWY0VDAUWuIjCADo6duxYG0Arx6+6WQIEFPeiDiWPe1wHHWX1q0H3vf/K2w9nwRcaB57Ktd+P0VPVRNoGXYIB1Nhp80aEV+2vd5ydlBx3I122tnYg6e+3YvACCnpBKv3dHfoKhswOGJcCidXBIZgT3yRAZYG3cSB05P0vL+z4dR7dU2Drvc6H+7yCutMnwdIg7wG49k4eXF+urInaZ2uxmeIGk4E8kqqsAHaw5j2b501a9V835eqXfvwJebE1Wvy1EEf852+64hY0hOPwP0IxRr5NkjT2SAxjepOmQIbwKQnmfMDYvGzYSqZueydkd8bWkM1sXVXB258Kx2uKJXR0PWc1zvMJD8e63DHnN7KKmoxre7UnBu59b4/RU9dbfn1tIa1sgTkRCivkqkRv+HEqu+j5fsVd9brETT9yPmU31n5aGG66ltvXFpja+YBO2pW+Ozcfe0SEzbeKzJc9UeL7bEZyGvpLIuiQeARVFpqtZR//1UVHvw455UU2cCta1Q/HIIQZ/9Go93Vx3CUwuisTcldIclpqaYyBPZ2JLodGTkh+aQdk7sMdqbkocx0yLx9oo409Y5J/J4g99re6TMPBH68+yd2J+aj0m/HEFiVnGD595eGYdHZu/C7z/dLGx987Yl4z+LDmDsnF2ITS8Q1q7d7UnKxecRCcgurjBsHSHYoekIM+qViE3d0PSEmWqEYmkNE3kiA4i6vHfkZBEe+GI7vCF0dK39GnbiW37gi+3Yl5qPOZHHsalRb7Edjy96trG/t5PQaF6Eeadvsj5ZqC/5rB/re6vPXIV4d5V5J01Wyi2pxP1fbMcHaw7jlcUHAi7r1vpnomBYWkNEQjROZvRIzy9DXEahsPacwolfx9X1boaIbjQrr1nHF7+lNZK5J0clFdUNJksyihHvSW+TG4+cwl1TtmJ6hLie03VxZyapWn/oVMBldV3Nkhv/6sS/RGOsic3E1A3xyC+ttDoUojq82ZXICezYnWs0h+cPwcI37uZXY9pVvn4ZBaVVGDZpIwrKxI1u5KQ/gce+2g0AiEkvwF0DzsF5XdrobpO97NaKyyjEUwuiAADJOaWY9MBVFkek7/YIq78nSBz2yBM5AA/hDuSAI6VREf7f+qNBk3gzE/OpG+JxqrDcvBXWk2LCVYnG9CT9TuuB93plU8op5m1Pqvv5R5U3aotm12GJyRpM5IkcwEm9kXrV3qzktISiscbRm/UZBpzZVeB6Ar2fLANvxtTio7VH8e8f9yte3so623VxJ/HPhdHYmZijuQ2n/+0odTizEDd/uBG3f7oFBSbMbUH65BRXIDa9ICTr2I3ERJ7IAb6KTLI6BL+qPdomuwrG6WPvl1Z6GvzeeE4wX3lwlceLV5ccwBNzdyMtT2xPrujSjNziSszcnNhkKDwZ5l5BUrqbbInPNjQOf9TkLOVVHvx13h6sijmBP365o+GTFp3M2znnevLrPUjPL8PhzCK8//Oh4C+ox659I1UeL97/+RBeXXIAuSWBa/Ht+h58ySupxE0fbMSoKVuxcGeK6td7nH5AMBBr5IkcQGStsSiyLOOv8/ZgR2Ju8IU1tu9ks7c2HBZydcyJoK+Zvz0Z3+5KBQAUllfhx6duVL9iFZtNzzaevO6o5teSb4ESNzVJm67SGgF/drIs43BmES7u3g7Nwo3rL0yrN3lbtMqx1e367bJwRzJmbKoZarKiyouP/zjQ4ojE+PTXeJRV1XRuvP5TLP58/QWKX7tifwZeWxqDmy/uhmkPDQo4xCSHnyQiUijiSBbWHzqF4opqq0NxhNiM4OOd/xx7JtnfnXQmMVEzSkagBMWMcyNZdv7B1KokT8t67bilX158ALd/ugVj5+yyOhTHmb8jue5noyans0JhufbOqGe/3Yui8mqsjsnE9oTAJWdO7wDSgok8EWny2NzdhrRbN468Ia1bR+vxZXXMCVz73nrd6/eVW1u5ja0YheX1n2IQlWzurJii6tX9nRyJ/gxFtPfDnpqbQbcl5OCkn5uMRdfxq/37suMJEOD8k2C/BH3c9a/CUA0m8kRkS27rWNH6dv6xMBpVHuWvtn67KQvAXyJnZPwLdqTgvs+3hWSvnVZ6t1SVQffQ6GXXPcClaTwZiIk8EdmSXQ+0Wpl1gFbT07n7uDH3N5iajBgxIZQNdz7T9h+T3rzoKzI2/MioPp6hGIaJPBHZkx2zKYGsmBCqcZL/9MJo5BRXYGdiDm77eBMemb0TUzfE40BaviHrr89fImdGZYFTdq0/fLEdj321C0XlVY4dgjZYqUigz6KwvAo/7E7F8WxxM2XXsuvmdOrnHJRD/uaciKPWEJGt1B7I3Pa9b9b7UbueXw+fwkuLDgAA4k8VY0t8Nj5aexSH3v49WrcI1xSDnmTEX2InMr8prqxGh1bNBbZojF1JNVdMJq89igHndvS5jN3zPj09/G/8FItl+zLQtW0LbHv1FrRspm1/dBuWhlF97JEnIlupPUa57VjVOOEy6mbP2oO83taVjLLjc/0612tGu9M2HAuyLnFrU7Mf+0vQ1h7M9Hty5OsVuk6kFMbUWF5JJVbHnFA9ilWgWJftywAA5JRUYqvgeQDs+vVixU3gWk3dEI+7p0Ui8pg1czRQDSbyRGRLoTI7pWiyDBzPLsGkX474fE5NO2bYePhUg99rE7tgw8zV0rKf+BtJRamySg82Hj6FEpsOvWr2SbAsy3hw5g78Y2E0nv0musFzekpr1Magxve71U9K1Nj6uJP41/f7sD9VeymaFnYY2ebYqSJ8tPYo9qfm4+FZO01bL48LTbG0hohspa60xmXf12a+nT/P2omcILNCGklJmlH7OTcexrT2c39wZqOZTW3kb/P3YEt8NgZf0Bkf3Hel4eurmS3Xzz0FotelYkctLK/Cp+vjkV9ahcOZRQCAjUeylK1H9PCTKpf/dlcqnripLy7q3k7T+orKq/DkvD0AasZ7T5p4p6Z2tKi76qarhE3f9o8/Wazr9Vq57bggAnvkiYhsQsRldVkG0vObjrVcVF6Nj1XMxqr1QC/iQOurN15kwhosxGDvYcvpMo+o5DyUVHjEBBUkHjVJm79ly6s82BKfhbJKMTF/vPYoZm89jsXRaULaM9vuJO2jNqXm6hvPvKSiGolZTZNhG3S2k8MwkSciW3Jbz0vj4/Oag5mGrMdfT2d6fhnWxZ1U0Y52ei79V3m8PnvjH5u7W1X8ehSUVcHjVbYFvEF2VDvtxn+fH4VHZu/C2Dn+SyEa7z+B4p+7LUlzLMJrwTVsaKu+Y0oqqjHsw424ZfImLNyZHPwFNmSn/TrUMZEnIlsKliBZTW2Pta+ljRhWz+rNprdkYo+fmVcrqr3467w9SM5puM12J+Xh5UUHkF8qrpSoqLwat368CeVVxve2K6Fmm3oDnIBsOlpT9rI7KQ+lldbW96vdT4Lt14nZJfjXD/uQVVShIyrl9PScz956vK707b9LYwVFZBe8pGA2JvJEZCtVHhler6y4R9Qqry6JUbW8r8Q/Jbe04TIC+rlEbTWrTwj8GT4poslj3+9JxYTVh1S1U+3xYv72JMzZetzn88ezSzAn0vdzRgm0zVcdOKGojbUKr1psT8hBtU1nXVUiu7hpwr4kOh1v/KQ8Mbbqxsn80iq/z9nhRlYl/EepbZsq/b636deSpZjIE5HtjJqyFdU2T+S/252qu41Avadaqb1SsNZPiY/WJEf5hFBi/bBHeZ22LJ9O+pYdxNsr4/wul+HjXoPGZvs5ETizLv2fcXZxpeIEPU/hlYknvt6DV3ydjJr0Z6e3tKay2vdJiFElayIZfQIRk1aAr7cloSDACYNeIt/B+OUHceX4X4L+LZFvTOSJyHbiThQaUnZiJV89bY3Lh4Tc7Kpy+fWHTvl+QseRuqI6eEmK1adpby4XU9KwUmFPuR5GXZ1aFBX85MeoKzOqS2tULPv+6kOKTsLsSO83QEFpFe6ethVvLT+I8SsOConJSKeKyjF3WxJKKz14J8BJNfnHRJ6ISCclva6+ljEiSRI2LrfJr3OLonLjekHN0Pjzm7ctCb/9v01YrCDpN1OgCpQZmxPxwnf7zAtGIL2VNStjMlB73rd0b7rf5Uyfa8DP40Xl9pyLwUmYyBMR6fCv7/dh8LvrsTpGfc+sx4Cj6S8OKC0ArL0lbndSLsqrjKkPn7xW+RCfRhC9XSevO4qjJ4vx7x/3GxKPUWUmuxQMLWnVfSCN11tW6RFSgkWhiYk8EZFGu5NysWRvOnJLKvGPhdHBX9BIsIP3pF8Oq27T14yuWmjNKxKzilFYZu9ethMF+mZ2DWTBDvOGE8wvrURClrISNBGJYlxGIab8Go/URjdpayF8+EkN9A2xKiwMDHx7Lf705Q54vbLwceRnbz2OYoEzEM/akoh//bCvyY36Wlm/FzgfZ3YlItJIb0ITrPR52sYEXe3rkVGgrcb4k/XxgiMJDVUeL3YkNp0Iyx+PV8adn21VvLzSPD7QcndP24oqj4zl+zOw7l/DFa/b53pOp9F2SOitVlHtxc7juaqvptVsu8Af7Dsr43Aivwyvj7pMR4Q1diTm4N1V6kaHEo0XLppijzwRkUZ6e8/sPFb+S4sOWB2CIzX+RP19wsdOFeFU4ZkrA//6YT/+o2Kbb0/I8TmDryT57uUUsadVeWpaiT/VdEZSrZSW1lhRejJ57RHcMz0Su45rnwFWjYyCckNObGbpHA1mW0I2bvt4E/70ZdOJ2uzO7sMYi8BEnojIIqFwkDFTZbUX3+xMsTqMoNbEZuLWjzfjxokb6q7qrNifoaqNKq+6Gn+libBVY6s3VhjkpmGje/L3p+ZjyoZj2JuSjz/M2G7a+vV2DiiNS82n/NDMnUJP3uozetz8Z75RX/LoNEzkiYg00nswt3GHvOGMOH5/tzsFry1VN1GXaEre1lMLogAA1V5ZeLz+7k+w667m72/of8sbDkVoSPwB/gBjMwqMWGMIMDYxf3dVHGZuTlR8YvpzrDNu/teDiTwRkUZuLq1xojeXiR03W8vHo/YlBWVih6v8YI3vG6RF1MgbwdcVgLiMQiyOtna4S6tq95WsNdBHpPSKinHvztgdqLTSg/dWHwqJBF0pJvJERBZhZQ1p5mff8TcjstUlM2pmGf3rvD1NHissq0K1R105UbCZk9NUTBql5kZkrcy8D8AuXz1aTyjmbU8SGIWzMZEnIjKBrwMne+RDQIh/xLVXrdTMMurrJt7/LDqA4ZMiUKJiKMWjp4oCPj9jU6Lf5xpfbUvKKQ34vDA6G7Z+FCCx6y+v8j1LNL86z+Dwk0REFmnSY2j1MZiCEt1rqrk5FfuK1ysrL63RFo3/9vzMMvp/6+IRm16IYf3OUtxWen4ZPtsQj1dv768o312wIxltWzTDrqRcXNqjvZqwg27eqOQ8Ve0pVVBa2eSxtDwxY7abQ+weNH+7efMyOBUTeSIijfSOuGDwgA3kYl9FJile9oXv9+HD+wcYF4wGMekFiEkvwLq4k6pel5anvBxmwY4zIxjtTclXtZ5gXl0i/qbqrOKKJj3/6fllGD4pQvi6GisorUJKbimu6NXB8JFk6nv/58Dj0qudvC0UO+pZWkNERBSA1yvjy80JmPjzYdWz1hpVm775aJbf5xrnYcv3Z6i42VVsvMFywph0laPDmJSpWXGS7avU5+0VB00ZpnbYpI24a+pWzNziv9xID1/7VbXHi18OqjuRq7XzeC7+MGM7MlTc5+BWTORt5I4re+CCrm2sDiMk3XbZ2VaHQA6k5lifmFXS5LFQrvM0s9dPq9qPZ/n+DExYfRhfbErApLW+R4XRKia9QPiEQ772K6tvdqUz1JwslVQ0rRGPOOL/JE55DA1/rx09acJqvfu38r/rb3en6lrTruO5eHlxw0nU7P+tIh4TeZu46aJumP7wYESM+42lcZzbuTXCw5z1p/DgkPN0t3FD367o1am1gGgolOjNRVccUDcJEFnji00JdT/XL9cQxdeEQ6LZdfhJuxJ502hmQTkSsmomVIpKzsXXBtZ9W39+7HsH8nXi/sZPsUFbC/Z+tsRnoyrIaEbxJwPf9Ox0TORtRpIkvHP35Zatf8tLIyw/mVBr2MXKb5YispPIY8YPaafHepX1y26l5uqBmUMI+jM9IqHJY9ZHpczd0yIVLWdEvirLMvan5vscNUerxKxiDP1gA0ZO3oRl+9Jx3+cCT9p8bAQb7H4+Gfl3cdX/1uIvX+3yu44Xvt9n2LrtgIm8DY2+qpdl65YkCed1cVZ5j4geCOt7MciJrB/qzVhP+hjPO5TUJgZ6PuXs4qajkFjBDicYSuxPDXxTqpElQj/tS8fd0yIx7MONyCgQk8y/tOhAXY3789+5O6Gsoe87MdjY/76UVnoQcSQLGw6f8vl8VlGFrpjsjom8Cfp2a2t1CC6nP5lydzpGRjFjkhiyzre7UrE9Qd9n/NKiA8EXMoFVabyTTnZf/H4/AMDjlfHJ+njd7aXmlmKPYcNUip0RuL5fDmZi6oZ45PsYCjO4hnva2yvisHSv8ll6B769Fm8tO1Nyo2bvWXngBCqqm5bZuH3iPQ4/aYIubVsgMbvpjW5+Oed7zxYcVtJPLjJ/B8c41sopf7YPztyBy8/pYHUYuonukB/3435l6zXqFMIBO5CvGWpFmR5xzOfjSq8uB/pc/j4/CgBwPLsUk/9wlerY6psTeRwAFI/jX1heja+3J+OJm/rifJWDfyzdm95kvoIa7s7k2SNvQyzzUEfUzbnc7kTkWoJvdl0UpbyX1QhF5eqGAbXC4UzjbrL0N+OpSIujtXzGvg+kardFTom4chiHVJVpxkSeHK9183BFyz0z4iK/z0mS5Po/drKnSh+Xgu3meHYJYtWO9+0ibjjJj0oRO8SlUkaU1iyOSsPIyZuEt0siiDmQijwcu/3QzkTehlxwzDDV2R1bKVouUMe9JLnjYE3Oc/OHGzB763FkFVWY0sum1pHMIoz4KAKjpmy1OhTS4fG57rlx+d8Ky3qU+HCN2HkBRNuTbM0JmJs45UZvrZjIm8Ddu5D1mH+Tk50srMA7K+Nw7XvrcSDNfr3eSmuh3cxJN2zqZfeJo0TnZNMjEpBTbN9RTcqr/F+x85Wgum1PFdHBZu89Wj8hibwkSfdLkjRFkqQtkiQVSpIkS5K0QGNb50qSNEeSpAxJkiokSUqSJOkTSZI6i4jVrup/edphxsNHb7jA6hAUC1O6vWywXYmcprDcuNEx3MjtSYMbFVfYv97elypPw73N67X+NOyt5QctjqApLUNaOomoUWteB3AVgFfb40UAACAASURBVGIAaQAu1dKIJEkXAtgGoDuAZQAOAxgC4HkAv5ckaagsy64f780O6eb40ZfDKztjVA7FebzG54gotIVSH4DLqxBcbdrGY+jWvqVl66+s9gqb5E7kfuj2XVpUac2LAPoB6ADgaR3tTEdNEv+cLMtjZFl+RZblWwD8H4BLALynO1ILKKnPstulW0mS8M6YK6wOQyi3/zETGSE5p9SYhu31lScME+GGnHAS5MTPzNeV+8nrjloQyRkeFT3fJWZeBXHg56uGkEReluWNsizHyzruKJAkqS+A3wJIAjCt0dNvASgB8IgkSa6fXckJX3x2ori0hohIAzuUOxql8WHb5TmP6yndU60+ebluwq+mrcvt+7Sdbna95fT/a2VZbnB3hyzLRQAiAbQBcL3ZgVEIcPGBmoj0cfO3wwIHlE/WZ3UCSv6pqdBXel+CiJNot49aY6eZXS85/b+/a0PxqOmx7wcg4KmcJElRfp7SVLuvl5JdqMHNrq4+bIhndg7+8HXn49KeHZBVWA5JkvDpr/qn8iYissIbyw7ikRt6Wx2G40Ul52LwBV2sDkO4LzYlWLRmGbHpBfhyc6KAltzNTol8x9P/+xt/rfbxTibEYrrwMDtdHHEWxWfsAc7KJSg/IXjvnivrfv7lYKayFxGRY+1Lzbc6BNO4vffSKPd/sR3H37/TtPXtOm7O+PITf1Y+zr7oXefuaZFC2vG6fJ+2UyIfTG2aFfQTkWV5sM8GanrqB4kMSpTWzc8k8qzyUEfx5gqyYbX8rau5uYeIzpixKRHX9+lqdRhEQpidK8adKDR3hQqI3gSijq9uP0zbqRu4tse9o5/nOzRazjGU/IG3ah5ufCAupfjEJ1CPvMaZXXnORaTdY3N3Wx2CKZzUyy06Uie8dQeESHq4/AO2UyJ/5PT//fw8f/Hp/60dX8kgrZoxkddK6ag1Lv9bJiKbqk1mMwvKMYX31NjOvO1JqPb4n0GVlLHrCav102QZy06lNRtP//9bSZLC6o9cI0lSewBDAZQB2GFFcEb7w7Xn1f3M0hp1RGwuCZKmniM3D0tHRGJ4ZRlhkPD0wijsTbFfvf2sLYno3bUtbr3sbNv3oBuRlH0VmYT+PToEX9BFbP4xC2X3fVov03vkJUlqLknSpadnca0jy3ICgLUAegP4Z6OX/Q9AWwDzZFkuMSVQkww6vxM+vG8ABl/Q2epQcJaFM8LpIgELn7wO53dpI7TZAed2xNgbLgi8aubxRBREbR5hxyQeAN5ddQhPztuDuAz71V2b5Z2VcVaH4Hgi82XO7KqckERekqQxkiTNlSRpLoBXTj98Q+1jkiR9VG/xXgAOwfcQkv8AcArAZ5Ik/SRJ0vuSJG1AzcyxRwH8V0S8Zgu0E70x6rIGvfGANcNPhknAzLHXmL5eXyQJePvuy5UvDwlDL+qGzS+NwOQHrhIWxzd/vR5v330FNo77jbA2iSj0OGXUjBmbxQ816PayBruxsnPJrsm3XUt+RBFVWjMQwKONHut7+h8AJAMYF6wRWZYTJEm6BsDbAH4P4A4AJwB8BuB/siybM96Sxcz+Q1zyjxvRo0MrnNOptbkrDkDNJgirt7DWP1df27z2oRbN/J/vuvz7gYgEcMr3RM13nr2DzSutsjoEW7O01F9kIi+wLbePWiMkkZdleTyA8QqXTUKAPE2W5VQAj4mIi5Q5t1NrdO/QyuowGlJxNqO0Tj3YF4NTDrZE5CzVDskkJEmyNhFUwKzx0/XYfDQLkQnZlqz7210plqwXsPfVl8VRabhv8LlWh2EIO41a414qM0QjOuT7dGtr7gr18rHNLureTsHLzP4ise8XFxHZw0wBs1OaYenedPzuk81Wh2GJsiqPkHZyiiswds4uzNhkzWcek65whG4DjpVCS2sEx/fvH/cLbc9OmMiHgE5tmmPdi8PQPNx3xq50+EYrPT/yYoy/y3fdfIPSGo1/+4G2QKDn2ItPRMF8GsJDTjrlO1LEVROvV8aORPtfMTCK0Lp2gW25nZ2Gn6TTRAxp2KJZGDq2bg6PV8b3f7sezcL9n7PZMY0PC2sYVf+e7f0u27rFmTH4q7z+rwurvexX+zEEehW/bIiICAA2x2dZHYJf6+NOYltCDm69rDtuvLCbITfjiexFd8oJoB0wkTfByP5no9/ZNYnoj1FppqzzvM6tseaFYfDKMloGmWzKjj3yzcKCx9SiWRjev+fKBu9P1JTOgLLRg/hlQ0REAPDst3sx8d4BVofh05Pz9gAA5kQex7oXh+EzA64Sie2R58FVKSbyJnhu5MV1PytJ5P2lj83CJLRoFobSSmW1fM0D9MI3WJ/98niEhwWP/ei7tzd5rNqj7Y8/0Daw4eYhIoeZsPqQ1SEQAQDu+GyLIe0K7dhiHq8Ya+RtKCxMwn2Dmt5dPbJ/d+x5/VZFbagpz7Hj7KSNz0FkWdkZenWA0ppAamZ2bdh+7blE4NIaftsQUXBfOuSGV9FC7RvShofTJqo0dngFw+OhNZjI29Qzt1zk8/E2LbRfRPFXKqKgisV0SnrkfQn0BaW2tyBYSZKWNomIyJ2KyquxJNqc8llbEng8TMopFdeYyzGRDyH+zpYD9cjPfvQa9Du7HZ675SKc18W8CaOaazy7CFQjf36XNpratOF5DhER2dD6Q6esDsEyIvu1XlsaI7A1d2ONvE35Sh6V3HypRaCceWT/szGy/9kAgJ/2Zahu+9renbE7KU99TBoT+eoAs5m0bdkMX/3lWqyOOYGknJKGcQVYXTM/w3YCoXfZmIiIyBdeobYGe+RDiL8TAaNOEADgh7/foOl1jUetkaHsS6Jdq8DnpiMu7Y5JD1yFy8/p2ODxQFuge/tWuKFvVwDAvYN6BQ+CiIgAWDFJH1mFNfLWYI+8S/lKTHt0bIWU3KZ1Z0benKP1RlqtZTCPXN8bE1Yf9hNL4NcG+gqa/8QQHDlZhP49OjR8DQ9SRERE7JG3CHvkXcrX39P0hwch3EfJih3vsr/4bP8TQAXSukU4/nTteYKjAZqFh+Hyczo2KfnhFxcRUWDs8AgN/JStwUTeZPOfGKJoufqzldbSm3Bf0asjtr1yCxY+eV2Dx+02IZSvaNQcB3xtu8ZuubR7g9/P69KGSTkRkWCyDPzxyx1Wh0Em4AmbNZjIm+zmi89StNzZHVrh9it6NHisNt++8cKuTZZvPKKMv9T87A6t0LNjK0XLOpW/75L69wIM63cWLu1R0+t/1bkdcX3fpttU0brYB0FE5NeGw6ew63iu1WGQCZjHW4M18jY2/eFBuPqddcgvrQIADO9XcxIw6YGr8Nev96BZuITX7uiPqOQ8jBrQE8MnRShq9+wODRN5u/XI+yIiYW7cC7/mhWGQZdmWE2IREbnB5xEJVodA5GpM5G1MkiQsfvpGvLXsIM7v2gYPDK6p/e7VqTVWPXdTXQKqtje5bctmePSGC/D19mQ8cv0Fmod6tCtfl/fuvbqXn3KlM+9dy8kCeyCIiPzLLCy3OgQyCY+H1mAib3MXntUOCxrVtAPaR4Op9b+7r8C4312C9q2a62qn1s0Xd0N4mITObVpg6d50IW02pvQ7wtdyWm+eDboufnERERGx1NQirJEPYaKSeAB4fGgfzH1sCAac2zH4wgZjck1ERGQuHnutwUTeYt0b1auLYkTZt5I2ff0h19b266HmC0Jrr4CWLyF+bxEREfF4aBUm8haY+tDV6NOtLZ4feTF6dWod/AUO9+H9A/DY0N662vA1/r0/ZvYKcLgtIiIKdRXVHry8+IDVYYQkJvIWGDXgHGwc9xu8eFs/q0MRzldae3aHVnjrrssVt/HPERcBAB667nwAQLd2LTCyf/dALwkag1GYxhMRUaibteU4hxm1CG92JSFE3eTy5E198PRvLgQAvHXXZRhxSXcMOLcjWjYLPslTXSzM5ImIiEyzODrN6hBCFhN5so3ObZrj9VGX1f3eslk4brvs7LrflZexMLsmIiIyDQ+7lmFpjUtJJs/XaqdSca2xaHkZh9tylr5ntbU6BCIi10nMLrE6hJDFRJ6EssPNn1pDGHrhmYm1rrLBMJok3qT7B1gdAhERkTAsrSHX8dVLfvPF3YK+7s27LkdCVgnKqjz49E9XK1uX9ectpEL39sYM90pERGQFJvIkhBkJreIK+UYL/m/05biiV/Ae9i5tW2DFszdBlmXFM+cyjyciIrK/ao8XzcLdV4jivndEAIyZEEqv//zukoDPi0qKG7fz6I29Vb1eaRIPAL+7vIeqtomIiMh8y/ZlWB2CIdgj71J2LPl44qY+kGUZzcLDMPHnw4atx8z33qVtC/NWRrrZ8QSXiIiM99byg7hv8LlWhyEce+TJNK2ah+OZWy7GU8Mv9Pm8qAScI8mQP2quthARkXvYYTAOIzCRdymz8xVb/XmYHMy5nVubu0LSLIx5PBERuQgTeVLMKTmQ2ScVM8degw6tWKXmBGbPr0BERPZgqw5HgZjIk1B6rlwFveylsG2zL5/179kB214daeo6SRv2yBMRhSaXVtYwkSfl/nvnZVaHoIgVf6vNmCE6Az8mIqKQ5Nb755jIk2K39u+OyQ9chfF3NU3oa3vB7fCH4tazblJmeL+z/D4XxptdiYjIRZjIk2KSJOG+wefiL0P7NHlORO785M19BbQCXNu7c93Pnds0F9KmKBPuudLqEFyvbctwv89pSeN5sYWIyPnc2snHRJ5s4Ymb+uBvw8Qk8g8OOR93XtkTl/XsgAVPXiekTVH692yPXp04yo1VtAw/efB/vzcgEiIiIv041AYJpeWM9+6B5+CNUeLq75uFh2Haw4OEtSdSmCShQ+vmSM8vszqUkKSld711C/89/ERE5Awu7ZBnjzyRmViibS0OP0lEFKJcmskzkSch9NSeKb0B0Q430pL9BUrWJX7jERGFJLfmEDysuZRVU9Hfc3Wvup/vuuocRa8JtT5St04TbRsBdqhQ29eIiMjdWCNPQlx+TgcAQPcOrfD9367H/rR8PDD4PIujsh+WdliLw08SEYWmKo87O9GYyJNu/c5uh/O6tKn7/bq+XXFd367KG3BBbsX80D4CfRT8nIiIyE2YyJNuDw05X9fr3dBLrbRahomktdT0yLdv2Qwv336pgdEQERHpw0TepZyULzK5tcag8zshOiXf6jCEE3V/yP63foswzgZFREQ2xptdyTHsfI+oE09GPnvwanRsba+Zb42mpkeeSTwREdkdE3kXuemibnU/3zmgp2nr1ZtfM12yxrmd2+DdMVdYHYapnHjCRURE5A9La1xk0gMD8MZPsejcpgWevLmP1eEo5obkqmUzZbN/uuG92l2gTcxRa4iIyE2YyLtIz46tMevRa01fr97UyA03uwLAvjdvw8KdKSiuqMbnEQk+l7HLe/3qMfP3Ezuwx9YnIiISg6U1pJvu0hqF2ZWda+QBoFObFvjniIvQo0Mrv8vYoUP4jit7YMQl3f0+f2mP9n6fe2bERUaEZBpf2/8P15xrfiBEREQCMJEny9khuTWT1SckbVqcuRDnK5TP/zwYXdq28Pna2om/7CzQ/uRrRBu7XCUhIiJSi4k8kYnscNISKIQf/n4D+nRri/lPDDEtHiIiItKGiTzZgA2yWxt56LrzcVH3doa1H+hkYkifLoatl4iIiMRiIk+66S0VUdpLHRYie+voq87B+n8NN6x9PaUkNr9NQRPZle+KiIhCQYikRmRnStPKmy46C93atQQA3DfImTco2qEeu/6Jk2x1wb4BrN/CZLV/jrjQ6hCIiEzBRJ4sp7RHvkWzMCx7Zig+e/BqvDPmcmODMtD7910Z8HmjE1FfN3y6SesWgUfVvfPK4JOlvfT7Sxr83qtTa10xkbn+enNfq0MgIjIFE3mynJpe6l6dWmP0Vec0GHnFbgKPmgJcfV4nzHt8CD578GrcP9j8KwtG5PF2GsLxxdsuRotw/19tQ+vNgOzLN09eh6eGNezRbd/KvvsbNWWHK19ERGZgIk+Wc3kHcQMSanrEh/U7C6OvOgfNwqx9880DJLxqfHDfACHt+DJm4Dmqlj+rXUv8f3v3HSdHXf8P/PXevd77XXK95pJckmu5XC49ISEBYkIgkJCEEDqE3qQoRUTAgnTBAlixoOLvK6goXRFEAbEAghhBUFEEpAQk5PP7Y+cue3s7O2Vnd3ZmXs/HYx97N20/+5nP7r7nM59y35mLddfHtokvzc8e9/9IRxVCLp8XIiIiMxjIU9L818o6dcxctKS66Uv00feaWovq4ki/g0PnNts/ZgrTfP7q6Th8XivOWdVtep/JZfkTJubSS+IJSztRnBupcb9sXfxmTz7sSuBvNopjWUG28UZERBmG94vJdX6r+7QS9CUb/5659xT85+3/4Us//4ut18zJCuHHJy/A7156w7DJCeBOQFtRmIPzV0/D7t0Kl/7oacPt9S4q9LK6ND8bP//wUvzzzXfRVas/qy3526/P2wsd5/3I7WQQEVnCGnlynd87X45n/F6tZEd1US4+ut80iykY/wKVRblYPKXGsWY2XlRakJ0wiOcQlf6XFeDyT0TexW8uckV/U9nY36t66lxMifMy/bok09Onx6vppvRjWSGioGDTGkqanbHIr9rQh2vueRbddSWY01aZglRlpkwIMDIgCa6y0zyIbeSJiCgTMZAnVzRWFOCTB85yOxkZYGJYbSnQthGVJ9OUaXJZnvFGJm2b14Kbf7HD9PbJNsFKJhZnHO8tQb9YJaLgYNMaojTyeoDR11SOdf31jhwrKyRoqyp05FhW2AnK/TgDLhEReR8DeSIfGmguR4nOJEbJNu+54qBeNFUUJHcQRGrYe6P6SgCRMeP7Y5Y5JZm3XV9u7f1evLYniVdzTiY05XJDblbY7SQQEaUFA3miNIptHhIv0Eo2+MrNCuG7x41gVmP8gPiA/uRnYXUiQFRK4YQlHeOWXbmhD+UFOckf3GGf2N9aYL5l2P6Y/E4KG5yo5srkL8gyUU4Wf9qIKBj4bUfksOGozrtVRbmOHvvcfeJPilQeNZlNrTYR0hkrpowtO2ROE45e2IYrD+5FT32p4euka4r7/JyJNaepasQy0h4ZJ39Wg/H7j9VgsUY+UySaofb8/aahT+dij8iv3J5Nm8hpDOSJHNZVW4yPr+3BfjMn4etHzhm3zsxPSEVh/OD/zpMW4OiF7XHXfXHrbIQECIcE12/qBwDMaizDjVsG8LE10/HRfafh3H2mYm2fufbtRuOmO9FkvL4sH1mh9H0FXX7gTADAzIYynLa8C8NtFfje8SNpe303JKqRP3x+q+66webyVCSHUqQol+NWmPXQOUvdTgKRo/jpJ0qBzcPN2ByneYXRLf9D5jShVacD6LTJJROWjYZpA83lePDDSxEWQV3pnpFl9p6emWP0d9cV45A5zcjJCmGopQK/2vEfrJ41OaWvWV+WP/b3Scs6cdKyzpS+XiYIB7j2saOmCM+98pbbyaAMU1Ps3MhbRJmAgTxRim0ebsLXHn4BCzqrMDkqmAQm1tB/Yv8Ztl+nPubYqZTMTKfbl7TjjBVTxvoLfO3IOfjDy29gZkOkmQdHiHGOXhzv5xp3p5uzUeY6eVknrrr7WbeTQeQqBvJEKXbxmh4cMb8NzQ6M9OIHhblZ4zr95mSF0Nfk38DSTXo18ldu6E1zStLnsBHjjsaFOWHs31+Prz38QhpSlHpBvfg109+HyO8YyBOlmIjoNpfJVOnq7EqppTeJllc775qxW4tpjYLbD3anITGUUkG9gCGKxs6ulDR+l9qX7DCOyc52alcy59xoXxYnSofdu1nSvG7n+x+4nQQi1zGQJyJH3L59niNj1DslqJMheVFnTZFjxzJzkSki2M0aCM97+z0G8kQM5ClplUWZN4EPJcdOZ9bexjJ85qBZKUjNHium1QIAFnRWGW7LOC2YRsuu0en3U4W8j96KJf/bxUCeJirOzQrUiF0M5MmWS9fNQEgiwwiu6TU3Njn5R3JNa+yPUf+5zQO4ffs83HTY7LSO0hM0XggM9Sb2Mls249XIz7QxWZhX+eGOlZ8uxii+vabWWt5n6dQaPHjWkhSkJjMxkCdbNg414dHz9sKdJy0I1JWv0zprik1tV5qfHXd5qnI+Uzu7hkOC3sYyZIdD+PyhA7r5AvgjUHGC3mzAgHt9LJxw1Ya+uMvNxnZTJ0387F2y1trwr9XFmT3U5aY5TbrrCnO8P9YFm0f5X11pLvadOcnSPgJvVEY4xbFAXkQaROQmEXlZRN4TkR0icqWIWBpXTkTmi8gPtP3fFZEXROROEVnpVFrJGZVFuQmngCdjh8xpwmBzOUrzs/Hlw4d0t0v0g+w1Tv32Tp9cikfOXRaomhc7to60WN7HC5/qFoORoBK9BwFw2EgrhlorUFOci28fMxcAUFNiLTC/9ahhS9un21l761/E+QHj+GCw2jFdRALVmd2RS3IRaQfwEIAaAD8A8DSAIQAnA1gpIvOUUq+aOM5xAK4H8DaA7wP4G4AGAOsArBKRjyilLnEizUSZIDscwm3HjeD9D3YjO8wbZIC1mpS87DDvCBnIzQpjVkMpfvu3N0zv4+mfQGXcRl4hMn/Bt4+Zi927le0KiQ4HO+k6ra+pDKUF+nesPvB4oHP5ATPw2jvvu50MSgOrd15EgnW3xqnI4XpEgviTlFJrlVJnK6WWAvgsgCkADINvEckGcCmAdwEMKKW2KKXOUUptATAI4D0A54lIZt/LJLLBKIhPdwuIZGZuNT42pZ2Hm9BYZTU+DepdRa8H8gu7qgMVrAXZ+x9YDOQhgeo/kXQgLyJtAFYA2AHgupjVFyBSu75FRIxmxKkAUArgT0qpZ6JXKKWeAvAnAPkAMrcKhCggkpmIZaS90sGU6Dt+cfvY30fMb03La2asDA940n2dofdyXg3p7ZzeXbu9PSOWQDK9WJND7nn6FUvbiwRrsjAnauSXas93KaXGfTMopd4E8AsABQCMGhO+AuBfALpEpDN6hYh0AegE8ISZJjpEQZGqACheZ9fl06yPHhCtsSIfZ62cgsGWiqSOY9aGoSactrwLxyxswyl7dRrvQL4wdjcpOL/jCb8Hpk4qibvc6zWWIpzUa9T2Je3GGwWIIFhNa5xoIz9Fe/6TzvpnEamx7wJwt95BlFJKRLYD+BqA34jI9wG8DKAewP4A/gBgg5kEichvdFb5u+cPBUJedgjvvh+5Zp7VWJa21/3swb1J7f+FQwfRXRc/qIhmtSalriQPjRX5ePE/OzG7ZU/f+uxwCCctYwBP/hdKEMnfuHkACz9174Tl9WX5eOn1nalMVkpFgjW3U5EZlnbX4rp7/+x2MjJGpI2826lIHydq5EcH3tXrSTW63DDiUEp9B5Ea/tcBHArgbABbEGmeczOA55NKKZEP3HbsCFZMq8XFa3vQXp2+lmbhqGAhk74jQyHBN44cxsVrpuO6Q/rdTo5v7TW1Bh/Zd6rl/U5Y0pGC1OhLqiLOo21rEtXIN1UWxF2ek+XxzvUCfGDzZJ+59xTjjTxiTmsFeuqNK0i8rNbiaFIC8XwfECvS8Uke/YoxzFUR2QzgZwAeBDAVkSY5UxGpyb8WwDfNvKBSaiDeA5HRdIg8J7qpS099KT5/6CC2DDe7mCLrUjk2fWNFAbbMbUFNSV7KXiPauv56TCpNz2tlDkFTRfygMJEtc9NbTnebaVnj0YBdT7yRm4zeohttiJ3sqxJpI299NBMgfn551SFzmjJ23g8ndNeVWL44D4Uye0QppzkRyI/WuOtNiVcSs11cWjv4mxBpQrNFKfW0UmqnUuppRGrlfwNgvYgsTj7JRJRIZ+3EL8HoWj87MYCfBk6ZWV+KrHDiN/TxtT1pSg2woLMKH5o1ecLyVT11ANy9g5KRp10nQ7waEIXjfLj8Xh9pZ4hBb57dxLw8qZsZG2Y32mgmI4EaztmJdzo6wkyXzvrRRqp6behHrQCQDeD+OJ1mdwN4QPt3wE4iici87HAI3zl2rq19Rzte5WWP/3rx98/NRBuHmnDzYbNx50kLUv5aXz1iDq7eOH6m046aIpy10l63IC+P+FCWYOx0v/LKEJpOptJOG/lEfQm8LNHbqijMSV9CUiArHILVy9LR/LDaJMernAjkR3vRrBCRcccTkWIA8wDsBPCwwXFGc7xaZ/3o8v/ZSSQRWdNhs/39ycu6cOOWAfz45IXjlvvpN1REDGtvwyHBku4aTJus3371gtXTnE7amJu2zkZpfvCC2q1zW4w38lFZBOLXyPvlLe47Y1Lc5SJiuUber4F8In5oK265aY3Y28+rkg7klVJ/BnAXgBYA22NWXwSgEMBXlFJvjy4UkW4Ria0qelB7PlBEZkavEJFeAAcicll2T7JpJiJjiX7zEk0YlZMVwt7T69BSFTt1hLkfUa98+SY7adb9Zy7GYSMtprZdOb3O8vEzJWbJzQ6n7bWWddcgP8f+62VKnlnlpzbfsT62ZrruOsvDT9rIpu66Ynzv+BHrO6ZRorflhyE6rb4DrzaRs8upRkTHIzIO/NUicruIXCoi9wA4FZEmNefFbP+U9hijlPoVIiPT5AN4VES+KSKXi8i3ADwCIA/AVUqpPziUZiJKI68GSanSXKk/R15V0fhbwldu6MVNhw06ngYrp8TO+ds2rwWl+dnYONRokA5nCkdeGi8aMkkoA5oDt+iMjpMsvaYhAmB1TL8QozTYvd4pyXNipO7USdROfpcPAnnLfSEC9lvjyMdfq5UfBHALgDkATgfQDuBqAHMtTOJ0BIBtAH4JYG/tOMsB/BzARqXUqU6kl4iMBa1Ww4pU/lBUFeXi+k3jh9HMyw5jabe1CbnM1NKm8if+ifOX44LVkdrU0ed0stPO36slPl7TmnS769RFhtvYSaZekCoCzGwYP6p1W0xzwJNj5pHwa9OaRO/KD01rrN5V8OdZ1ufYZaZS6kVEgnAz28bNFnZwAAAAIABJREFUZxX55r1FexCRxpXfn0RNa+yMWmNyu2SbrHjdQ2cvRXZY0FlThGdfeQtzWs3PhLusuwZ3P/0Kpk0qcX14zOhRIzKtptxvP/R2AlSnP2XpHpfeTEVD7HscGwvbR18xgkhn5wP6G/Ddx/42Yf2u3bsn7uQxlpvW+PSCTU8G3JAjIiM1xe73vk/2u9FPX64CZ+5YxMuTnKwQRARfPWIOLj9gxoTa+ViVUU0PrtvUjy9tHcQ3jxked+xzVu2ZyMmtyXCy0tyOe35n1djf02M6HBfkZHZTCavijVpj9HnzfDBrozjZrZH3Ql5dsn/84W59UCFvOZL30U+NKQzkiTLUFQfNQlZI0FFThA1DTWl//cSdXW0cz3ZKgqmuNA8Hz25CZZH+RVxNcS6+fPjQ2P952WEsm1qLkrzxo9UMt1Xgc5v6cfHaHtuT8lg9f7Hl57vHOd9h8BtHzdFdd9bKbgy1VKC7rhjXHtKPm7fNHlt33aa+uPt49WKzt9Fw4vQJrLY79oLYszfhf2+e3oRG35PeXa+1vRPnl/Aau51d/VfC4/NXtQSRj6zrb8DiKTUozc92ZVSKRK/4yQNmYtstjwIAPrH/DHPHM/kWfBhfTLCwS2+UXfOOX9yO05Z3aeMsJyYiWBVnGD+9U+LEKYi9YzHLRrBpZKS9SnddSV42vh01F0JrVSFu3z4P2WHB9Ml68xd6U+uEEaKMef1zZqsPhI1IXkQ8fQEw2FKB2594Oa2vuc+MOtz5u384drzYi86QJL7T4OXzZQdr5IkyWEVhTkYOLbd4SjWu39SPT6+fhfWDDab28VPn2WmTS5Nqy3/FQbOSToMITAXxTpo6SX9MfDM2zUn/naVovY1lvgvi7fLypF+AvSYjQQvw3OL0d31sUf3swb0Jt8/An8yUYiBPRHHF1l5FfzmLCPaZMQkHDjSYngrb6z+i3zx6GDMbSnHsonYMNJfbPk5uVmjC8JJe0VhRgEv278HS7hrDbeOd77NWduPQuc04dlG74bZuSHcyDuhvQEtlAcIhwYJO/bsLZnz9SP1mRvG40XbayaZLWWHrx/LrqDVet9/MSThxaYfu+thKE6PZar3aRM4uBvJEFFewvgqNDbdV4v+dMB9nr4rMZWel1ikn6mKnOM/+bKvRbaFXTo8/42WqjP42bprTjJsOm514Yx2l+dn42JqesTwMupwswc9OW4RHzl2GAwfM3dnSM6/D2oWAU6ND5WeH8en1yd9hsmJmQ+mEfiDxiIxvdjSj3t7dGI/fvEg/iz8eVUW5mNtWqbs+9qLT6Ls3aL9dDOSJKKN45UfTSpvv6I6WN2zWH4VmqCXxUJPXHtKHQ+Y04bJ1MzCjIXVNRNz4IdzoQIfuZINTNyryssKROzTpri12qkb+txesSPoixKqLPmRuXgKlgM9t7kdlYQ7qSvJw2QHm+vOQvlSd64S16NZ7uwYKO7sSUVyx36t24owFnVV48Nl/o626EA3l+c4kLENcuHoafv/SG3jnf7tw8GAjrr7nOd1t53VU4Y6T5iMrFMKUumLd7T62NnGA0lBeYLpzcTIc6exqsbxcvGY69plRhyf/9gY+9ZNnHEiBtxj1hZlcmocTl3WiMDcLJ936eNKv59QFs5Xx452Kr/SSHq/MddeV4KFzliIsMtanxOoFX7LXWFccNAunf+e3KamkMKqddvolP7LvVNz2m4nj1Uezk12Jin9sZ9fY85GXHcK77+8ZL99P/bHMYCBPRHE58WV4zcY+/OypVzC/o8p37RYri3Jx92mLsFsp/PSP/zTc3kwny5pidydxAvTvNKT6xzErHMKCzmrsePUd28fw8g+4UY18U2UBNg414WcmypoZXu/sasZoluZm2Z+QzIkSNauxDA+cuQS7lcKiT91n+zjhkFifqdWF82x3dCA9ehN7jfrl2cvQd/FPx/5nZ1ciIoeUFeTgwIEG1FmYZdRLM7uGQpL2kWNS7dqN8cdYt8pu/DAcNZOt1U7BSTetsRG2ffe4uVjUVa07IU8i0XlkdnQqs+O/Gx3NO58ydzmRT0pFOoo3V1ofJjRavEnVjGJmr5znhC1rDMp87IRoo8cKwLUqAAbyRKTDrQr0ykJvjujiF40VBY4cx26b787aYnx8bQ/2mznJ8kgsbhhorsCXDx/CpjnNlveNnsQndvZZPU61bU9FjfxH95vm+DFpj+4EzfL07HZweKLmSnPfDU41rVnXX6+tM+jcGtsM1MN35uxgIE9EpqTrq/Gj+01Drtbu9vpN+h1DM4lTFz1++PlprSrEhaunWWo7HWvzcDOuPaQ/YX+ClEjjCcjJCo0bcm9yWT4+vX4WVvXUJdzPqQA8FcNPHjy70fmD+oBT3w/nrzbXyTeaU+e5raoQXzh00JmDxRGvac1op+YvbLX2uj5rxWmIbeSJKKPUlebhobOX4vWd76O9usjt5ARSd10xHn/h9XHLYn8c9505CXc8+fcJ+957xmJH09JRU4TnXnnL0WOadf+Zi5Nq06zn1qOG0VRZgMqYpkMHDjTgwIEGtJx9h+6+7TXOfCZcaSOf5gDLiZpZJ5LsVFZX25h/wqmzfPfpiyAieOOd9w23tRNIx9tldKjexbEzYRsc32/9sYywRp6I4nLzu7CyKJdBvIvOXjkVTQZNbK7Z0Icfnjg/4XCaTvjCoYNYP9CA4tz01ztZ6dsBAPM69MfCjja3vRL1ZfZGceqqLcZJyzpt7RstFXG8+x1oU/OlVVPifid0wN53slPnZDQ4NtMPxV7TGv29jALz2LXBCuMZyBORjtjarKDVcgRZaUE27jOoWQ+FBD31pSkf/7y1qhCfWj8L6wdT32wj2ba2n1k/cer4hvJ8nLBEf9ZKO05b3oX8qPb1g3FmGjY6LWY7zVrhdhifKiV52bjy4F4smVKNbyTZb6O92n6H1/wc66PvuH5tZZKV7xHDCaEC9lPFQJ6IKKgS/ODFjgShxyNxQlrUleZh5fTxbdzvP3MJWqqSG60EAE7dq2vc/98+Zi7mtlXixKUdmBNnVkyjAC4V5y03iX4RdkXf2Zjbbu6OiJ3gdm1fPW7eNoQRizPoxirJN56RNjss42akHVWYM/GulNGn1OkLNjMXt/aGn7STGp1jBaxOnoE8EcUVtFqNZPQ37akRrSn256g7euUhk2r8JpU6O+mYnc9AT/2e0WfCITE9rGS00enqWyoLcP5+03DtIX0TgvUZDaW49ehhnL5iivVEIjU18rlZYVy6Tn/CslQEWDcdNhuzGkqx74xJ2DrX+shB6bEnr8MGhWr7knY8cNYSVBXlTFhnpzw63qk5Rb8LTv7ecBx5IiIEr51hMmpK8nDD5n5sHGr0xJCJfvKFQweRHRbUFOfilL2SazvuRJk/amEbZreUo7YkF98+ZtjWMT63uR9XHtyL244bweHzW7HfzMmm9lvaXTP297r+hoTbmgnw2mw0A9k41GR5n2Sam0ypK8YPTpiP6zb1e2JOB6MmJH2N5ZhUmq97fr5//Mi4/43HkXc2ki81cUfB6Tbyhq8nsc1AR//KoFqGFMr8Uk9EGYGBfWIreybh0nUz0Vmb5iETk+HI71x6fiy3juypbY0OFpdPq8XD5yzDzz+8dGyUC6fYKfO5WWF859gR/PLsZRhorjDeIY6yghys7au3PCHWpetmYP++ehy9sM14KEid07auvx7hkGDapBJsm9dq6fWtGmguR1dtEa7fNJDS10lGvPhynxmJhwi1erx4ovtAROtrmtgfIpFU3DE7Yr5BubD4wRFJLpCfeLxg/VoxkCeiuIL2ZZgJMjnL9ZpFdNSk58KlubIQXztiDj6y71Scs0/3uHWVRblJjVs/yskyb7aPgZNqS/Lw2YN7ce4+Uw2b9NSWxr9IOHiwEb8+by/88MT5hs1A9Hz1iCG0VhVi49D4i4nYw333uBH85JSFcduDJ2I2OE3V5+majdZGaior2NNMZtnUmgRb7vGxNRPHjLfVtCYFEwYU2uh0a8TRNvIZ/D2aCgzkiSiugH0XBpMDJ7mjpghn7j0FfU1l+MZRqW1WNL+zCkcuaEOJwzXvevx8MXv9IfFrwUUE5YU5SV2ILOisxr1nLMal62Yabisingu8zPZ7aKzIx2nLu8bdWdk2rxUfmjUZQ60VuH37PHzr6PjNr9qqi3D36YuSTmsq7pcZHVPvor+xQr8Pi5NFYLLDfWUyHSeEIiIKqIWd1fjeYy8lfZztSzqw3eEhFim1ZjSUGm7jdICtV5Nu9DJTaovxzD/fNPUabdWFeP5fbwMAFk+pNtg6tR48a+mEZdnhEK7e2Gdqfyfm0khFp+apk0oSro8tN23VhWgoL8DSKdW48P/+qLOPheEn42x61YZenHXbk+ipL8WHZpnrU+IXrJEnorgmjKntsVozMpYpP3iZUvOdGanwL7un+QuHDpre9ktbZ2P/vnp8bM10zGwos/eCNpy2vAvddfaamUUHxoMt1trAG0lFG/lVPXXYd8Yk09vfc/pifOXwIWQnaP6WbEu0Nb31ePz85bjt2Lljd5MyaUStVGIgT0QUUKGQ4My9TQ5fGMAoVwCcuHTPnQY3ZpdNt+hgO12n3OhCrqky8SzD0VqrCvHZg3tx6NyWJFOVWPSY+UMtFUnNtvv5LQM4bnE7vnHknHHt6c3oMuhcn4oaeRFJeFdB72wmSooTF/MFOVkZUymQTgzkiSiuIH4hUrDFK/LHL+7AxWt7cPO22YZNCvSooFQN2uTFb5qvHjEH2WFBXnYIn1pv3BcgkcaKAnx4Zbfpyaa+efQwhlorcM6qbrQZNL8xE8hPLs0z9brJSpSSZGrkvVh+nOT/6gUisu3kZZ34/APP46iFbQzsKXBEgPycMLYMR4a+/Ny9f3Y5RemVKEgMCXDRmh5Lx9ML1jL5q0UvbUOtFXjo7GXIyQqZGlvdScNtlfj2MXNNbWvmGtLWTKwJj2c9MVaGnywKwJ0xK1gjT0S6Tl3ehd9duAKnLe8y3piS5sbU4pkcRGUcm3nl1YvgodYKHDzYiElxamwfPW+vsQscs9qqisYmmdpraq0jaUy1RIFwdXFuyoP40Q67I+2VyM1yfthHwP3vADPfe6drv0ELOqvQU2/cUTtIeFlDRAl5YbZEv3B6FkYneTMUtSY2oJgQgNs8PV5uWnP5gTOhlML6G36JX//1tbHllRYnqwIiAeO3j5mLh59/FYun1EQtD0LpsufzWwbx2Auvoa/JXsfdI+a34vr7Et9JsjMZk9Ml2mi40xOXdWLjnCZUFlrrQxAE/IUmIiJCZl9IuUlEUFviTDvqqqJc7Ddz8oTmEcu6zU2UlG5uX2PkZIUw3GavNn5WQ6mpCy4779FOtiTbRr6qKNfSRV9QPs0M5ImIMoQrTWsCUddOZumVhvNXT0NBThghAW46zPxwkGbduGUA3z9+xNS2VmeCDSqjEW1G2fkGCIUElx8wA911xTh7VbfxDkjcTMnOXQGKYNMaIqIMkZ3FHzPKTLUleXj43GV4691dmFzm/MyZWeEQ+pr0x1D/3vEj+OKDz2NlzyRUpKB5xepZk/F/v30Z9WX5eOn1nY4f3w1ma6TtNm06eHYTDp7dhFffeg+X/ejpPcfTuTRI1MSskB1YbWONPBGRi05b3oXcrBCOWtCKgpzM/TFLZTtmXr6448uHDyHLwrh/JXnZSQXxZorQYHN53L/7m8px/aaBlE1i9skDZuLGLQP4wQnzxi0Pwh0ro/Ny87bZKU9DUW6W7Q7QQa/MZyBPROSik5Z14vcX7Y3z9p3myusn+hE8OWqSm1P2sj/hjZFmCxP+kHMWdVXjoXOWjluWyqDITJ/fKw7qxeTSPNSX5eOKg3pTl5goV23oRX5OGHtPr0NVTJvyIPSbOHCgIe7yyaV5uOf0RVgyxZn+C3o5OVrmjlvcZuu4QW+Wk7nVP0REAZGdoSMDHbe4HfVl+WiqLEC7wcQzVn3z6GFsvelXKMgJ46IPTXf02Kni1aDu5m2zse3mR+OuqylOz2RAZjVVFuDBD0cuLsLJzBJkwZre+rS8TrqZHSzpiPmteOm1nXh95/u448m/jy2vLc0znHAqHr242snBm/bvq8f3H38JK6bVIi87NcNyegUDeSIiiisvO4yDZjem5NjDbZX41bl7ITc7FPgf4lRzqkY1WWYrTlMdwHt5ONBUyM0K45L9ZwAA7njyDpdTY84VB83C9iUdaGPHZzatISIKMjdvSpcWZDOIT5PyArMTFwW7mUIsv7aRHx2Xvj9mfPrDRlrG/j52UbupY5m9LHLy8klE0FFTZDj+fBCwRp6IiMiEs1Z2Y/0NvwQAnLqX+dmOWf9LbkjUFOymrbPxwLP/wsLO6nHLT1vRhZL8bFQX5WDFNHudT1t0asmN74Q4G5QH5c4LA3kiogALeD8xS2a3VOCGzf145c33sH4gNU2OUoWzp/rfx9f24CO3/97UtuWFOXH7BpTkZeO05eYvUuPZNq8FP3jiZbzw6tu49pD+seVGs9OyiNrDQJ6IiMiklT2T3E4CZSi3K4A3DzePD+RdSk9uVhh3njQf7+3aPa7p3EBzBbYvaccjz/8Hv/7ra2PLR1vHzGooGxvHf98Z/JyZxUCeiIgIQFaI3cZYK0pOEJG4/V/O3DsyC+wVdz2Dq+95DjnhEI7R2uKHQ4LvHjeCR/7yKpbZHFM+iBjIExEFmNlp3IMgJyuEj+w7Fbc8tGMsuPCLTIjPM6XjqNs15+m2rq8e33v8JbeTMc72pR3orC1GZ23RuLH760rzfDscaKowkCciCrBFXdU4cKABj/zlVVyydobbyXHdkQvacOQCexPT6ImeobQ03+zoMc5iTbs9Xs630euVC9dMR252CLf+6kVX0xMtNyuM1SmapVfPJ/b35/cbA3kiogATEXx6/Sy3k+FrbdVFuHTdDPz82X/jhKUdbieHAqYkLxvrBxszKpB3wyFzmtxOQkowkCciIkqxjUNN2Djkz0CCiNzDnj1ERES+Z66NiNMtSea0Voz9vaCzyuGjp8ZZK6eM/X3OqqkupiQ56RpHPVP7HGRoshzHGnkiIiJKias29OHWX72A4bZKVEZ1asxkh89rRUleNioLczDcVmG8g2akoxLP/PNNAMAUdiKnNGEgT0RERClRV5qHU5OcYCjd8rLD2DzcbHm/01dMwR9f/i/e2Pk+rtvUb7xDGmVqrTklj4E8ERGRz3l59BWvKMrNwreOmet2MsYwdg8GtpEnIiIiIvIgBvJEREQ+Z7ZCXlh171Osn/crBvJEREREPsN28cHAQJ6IiIiIbCkv2DNbcXEuu16mGwN5IiIinxtsKR/7u6OmSHe7kjwGYmRNVjiE27fPw9EL23DbcSNuJ2dMUO5I8BNLRETkcxd9qAdP//1NvPv+B/hczNCIZ+49BZ+56xms6pmEtmr9IN8vAhLfpfV99jaWobexLI2vSKMYyBMREflcdXEu7j59EXYrIBwa36F1+5IObB1pQRGbRfhWUGqng4ifWiIiogAQEYR1BqVhEO8/itF7ILCNPBERERGRBzGQJyIiosCY11E19ndtSa6LKUkf1s37F++lERERUWAMNJfj7FXdeOyvr+GMvae4nZyUYfAeDAzkiYiIKFCOXdTudhJSLsRZegOBTWuIiIiIfODcfboBACLAh1f6926DGUHp7MsaeSIiIiIf2DavFU0VBWgojzzI/xjIExEREflAdjiElT2TJiwPSOV0ILFpDRERERGRBzGQJyIiIvKx9upCt5NAKcKmNUREREQ+VlmUi6s29OInf/gHjlrQ5nZyyEEM5ImIiIh8bk1vPdb01rudjLSpKsrFf9/dBQDICfu3AYp/3xkRERERBdLVG/swOpT+LYfPdjcxKcQaeSIiIiLylZ76Ujxw5hLs2q3QWuXfPgIM5ImIiIjIdxor/D+WPpvWEBERERF5EAN5IiIiIiIPYiBPRERERORBDOSJiIiIiDyIgTwRERERkQcxkCciIiIi8iAG8kREREREHsRAnoiIiIjIgxjIExERERF5kGOBvIg0iMhNIvKyiLwnIjtE5EoRKbdxrBki8hUReVE71isicr+IHOpUeomIiIiIvCzLiYOISDuAhwDUAPgBgKcBDAE4GcBKEZmnlHrV5LEOA/BFAO8A+CGAHQDKAPQA2AfAV5xIMxERERGRlzkSyAO4HpEg/iSl1DWjC0XkCgCnArgEwLFGBxGRYUSC+N8DWKmU+kfM+myH0ktERERE5GlJN60RkTYAKxCpOb8uZvUFAN4GsEVECk0c7pMAwgA2xwbxAKCUej+51BIRERER+YMTNfJLtee7lFK7o1copd4UkV8gEugPA7hb7yAi0gBgAYBfA/iDiCwBMABAAXgCwL2xxyciIiIiCionAvkp2vOfdNY/i0gg34UEgTyA2VHb3wNgccz634nIOqXUc0YJEpHf6KzqNtqXiIiIiMgLnBi1plR7fkNn/ejyMoPj1GjPBwGYCmCdduwOAF8FMAPAHSKSYz+pRERERET+4FRn10REe1YG24Wjno9USv1Q+/+/IrIVkeB+EMABAG5NdCCl1EDchIi8+tRTTxUMDMRdTURERETkiKeeegoAWlL5Gk4E8qM17qU660tittPzmvb8HoA7o1copZSI/ACRQH4IBoF8Av/duXMnHnvssR0290/GaLOep114bS9jvtnDfLOH+WYP880e5pt1zDN7mG/2JJtvLQD+60xS4nMikH9Ge+7SWd+pPeu1oY89zps6nVpHA/18C2kbRynVanffZI2229e7W0DxMd/sYb7Zw3yzh/lmD/PNOuaZPcw3e7yQb060kb9Xe14hIuOOJyLFAOYB2AngYYPjPAng3wCqRKQ2zvoe7XmH/aQSEREREflD0oG8UurPAO5C5PbB9pjVFwEoBPAVpdTbowtFpFtExo0go5TaBeBG7d9PRl8UiMgMAIcB2AXgtmTTTERERETkdU51dj0ewEMArhaRZQCeAjAHwBJEmtScF7P9U9qzxCz/BIBlAA4FMENE7gNQjUgH1zwAp5sZfpKIiIiIyO+caFozWis/COAWRAL40wG0A7gawFyl1Ksmj/MOIoH8RQAKEKnh/xAiFwn7KKWucCK9RERERERe59jwk0qpFwFsM7ltbE189Lp3AFyoPYiIiIiIKA5Rymh4dyIiIiIiyjSONK0hIiIiIqL0YiBPRERERORBDOSJiIiIiDyIgTwRERERkQcxkCciIiIi8iAG8kREREREHsRAnoiIiIjIgxjIp5iINIjITSLysoi8JyI7RORKESl3O23poL1fpfP4h84+IyJyp4j8R0TeEZEnReQUEQkneJ39ROQ+EXlDRN4SkUdEZGvq3lnyRORAEblGRB4Ukf9qefI1g33SkjcislVEfqVt/4a2/35236uTrOSbiLQkKH9KRL6Z4HUs5YGIhLVz8aSI7NTO0Z0iMuLE+06GiFSKyJEi8n0ReU5L3xsi8nMROUJE4v4WBL28Wc03lrc9RORyEblbRF6MSt/jInKBiFTq7BPo8gZYyzeWt8REZEtUXhyps03Ky0/K804pxUeKHgDaAfwTgAJwO4DLANyj/f80gEq305iGPNgB4HXsma03+nFGnO3XANgF4C0AXwLwKS2vFIDv6LzGCdr6fwO4DsBnAbyoLfu023mQIG+e0NL4JoCntL+/lmD7tOQNgE9r61/Utr8OwKvashO8lG8AWrT1T+iUwQOdyAMAAuA7UZ/tT2nn6C3tnK1xOc+O1dL2MoCvA7gUwE3aZ1MBuA3aBIEsb/bzjeVtXBr/B+BhLb8uA3ANgEe1NL8EoJHlLbl8Y3lLmI+N2uf0TS3dR7pRftKRd65ntp8fAH6inbwTY5ZfoS2/we00piEPdgDYYXLbEgCvAHgPwGDU8jwAD2l5tiFmnxYA72ofpJao5eUAntP2met2Pui83yUAOrUP+mIkDkjTkjcARrTlzwEojznWq9rxWpJ532nOtxZt/S0Wjm85DwBs1Pb5BYC8qOWztXP2CoBiF/NsKYDVAEIxy+sAvKCl/QCWt6TzjeUtqqzoLL9ES/v1LG9J5xvLW/z3KAB+BuDPiATOEwL5dJWfdOSd6xnu1weANu3k/QUTfwSKEbkaextAodtpTXE+7ID5QP5wLc++HGfdUm3d/THLP6Ytv8jK8TLtAeOANC15A+Ar2vJtcfbRPV4G51sLrP/QWc4DAA9oy5dYOV4mPACcq6XvGpa3pPON5c34/c7S0vdTlrek843lLf57PBnAbgALEbkzES+QT0v5SUfesY186izVnu9SSu2OXqGUehORq7MCAMPpTpgLckVks4icKyIni8gSnTaPo3n24zjrHgDwDoAREck1uc+PYrbxsnTljV/zc7KIHKOVwWNEZGaCbS3lgZbnI4icgwfN7JNh3teed0UtY3kzFi/fRrG86VutPT8ZtYzlzVi8fBvF8qYRkamINEm6Sin1QIJNU15+0pV3WcnsTAlN0Z7/pLP+WQArAHQBuDstKXJPHYCvxiz7i4hsU0rdH7VMN8+UUrtE5C8ApiNyt+MpE/v8XUTeBtAgIgVKqXeSeRMuS3neiEghgHoAbyml/h4nDc9qz11JvA+3LNceY0TkPgBblVIvRC2zkwcdAMIAnldKxQvqMjbfRCQLwKHav9E/TixvCSTIt1EsbxoROQNAEYBSAIMA5iMSjF4WtRnLWwyT+TaK5Q1jn8uvItLs7VyDzdNRftKSd6yRT51S7fkNnfWjy8vSkBY33QxgGSLBfCGAGQBuROSW4I9EZFbUtnbyzOw+pTrrvSIdeePHMvsOgIsBDCDS9rEcwCIA9yLSLOdu7Qt6VCrzORPz7TIAPQDuVEr9JGo5y1tievnG8jbRGQAuAHAKIsHojwGsUEr9K2oblreJzOQby9t45wPoA3CYUmqnwbbpKD9pyTsG8u4R7Vm5mooUU0pdpJS6Ryn1T6XUO0qp3yvZ5l6mAAAE4ElEQVSljkWkw28+Iu3XzLKTZ4HIZ6Q3bzyTl0qpV5RS5yulHlNKva49HkDkbtgjiNSYxB2WzOjQFrbNyDIoIicBOB2RkRS2WN1dew5ceUuUbyxvEyml6pRSgkhlzjpEatUfF5F+C4cJXHkzk28sb1GJEBlCpBb+M0qpXzpxSO05leXHkbxjIJ86RjXBJTHbBc0N2vPCqGV28szsPv+1lLrMk468MdreqHbBM7TbnF/U/rVSBuPlgec+6yKyHcBVAP6ISCes/8RswvIWh4l8iyvo5Q0AtMqc7yMSZFYi0tFvFMubDoN809snUOUtqknNnwB81ORu6Sg/ack7BvKp84z2rNf2qVN71mtD73evaM/Rt/1080z7oLYi0rHseZP7TNKO/zePt48H0pA3Sqm3ERmnuEhbH8tvZXb0FvVYGbSZB88B+ABAm3YuzOzjGhE5BcC1AH6PSDAab2I2lrcYJvMtkUCWt1hKqb8iciE0XUSqtMUsbwZ08i2RIJW3IkTKwVQA70ZNAqUQaZ4EAF/Qll2p/Z+O8pOWvGMgnzr3as8rZOLsf8UA5gHYicjED0E0V3uO/mK+R3teGWf7hYiM8vOQUuo9k/usitnGy9KVN0HJT2DPiFHPxyy3lAdanj+EyDlYYGYft4jIhxGZxOQJRILRV3Q2ZXmLYiHfEglceUtgsvb8gfbM8mZObL4lEqTy9h4ikyzFezyubfNz7f/RZjcpLz9py7tkxq7kw3As00BPCIXIKAMVcZY3I9JbWwE4N2p5CSK1CFYmBWmFRyeEinkfi5F4PPS05A08MGGKxXybAyAnzvKl2ntRAEaSzQOYm/SjxOW8+qiWxl/H+1yyvDmSbyxvkXR0A6iLszyEPRMb/YLlLel8Y3kzztMLEX8c+bSUn3TkneuZ7OcHgHYA/9RO4u2ITO99j/b/MwAq3U5jit//hVrB/hGA6wFcjsiU5ju1PLgj9ksIwFrsmab7iwA+iahpuhEzjby2z4naetPTLGfCQ3uvt2iPH2vp/XPUsk/H2T7leQPgM9r66Cmo/60ty4QpzE3nG4D7EAkQvqO9l88iMtyr0h4fcSIPMH4a7qe0c5MxU5gD2KqlbZf2fi6M8ziM5S25fGN5G0vfKYiMs383gM8j8tt3EyKfUwXg7wCmsbwll28sb6by9ELECeTTVX7SkXeuZ7LfHwAaERmC8e8A/gfgr4h0lkpYs+OHByLDYN2qfRm/rn1B/QvATxEZg3nCF7O23zwAdwJ4DZGg/3cATgUQTvBaqwHcD+BNRGbMfRSRMXRdz4cEaR79gtF77HArbxAJYB7Vtn9T238/t/PMar4BOALADxGZYfgtRGpAXgDwLQALnMwDROblOFU7Jzu1c3QnYmrEMjTPFID7WN6SyzeWt7G09SAS4DyBSJCzC5EOfY9qeRr394/lzVq+sbyZytPRz/CEQD5d5SfVeSfaixARERERkYewsysRERERkQcxkCciIiIi8iAG8kREREREHsRAnoiIiIjIgxjIExERERF5EAN5IiIiIiIPYiBPRERERORBDOSJiIiIiDyIgTwRERERkQcxkCciIiIi8iAG8kREREREHsRAnoiIiIjIgxjIExERERF5EAN5IiIiIiIPYiBPRERERORBDOSJiIiIiDyIgTwRERERkQf9f1KWrctWuK27AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x2ad8a057f60>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 250,
       "width": 377
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(losses['test'], label='Test loss')\n",
    "plt.legend()\n",
    "_ = plt.ylim()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 获取 Tensors\n",
    "使用函数 [`get_tensor_by_name()`](https://www.tensorflow.org/api_docs/python/tf/Graph#get_tensor_by_name)从 `loaded_graph` 中获取tensors，后面的推荐功能要用到。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_tensors(loaded_graph):\n",
    "\n",
    "    uid = loaded_graph.get_tensor_by_name(\"uid:0\")\n",
    "    user_gender = loaded_graph.get_tensor_by_name(\"user_gender:0\")\n",
    "    user_age = loaded_graph.get_tensor_by_name(\"user_age:0\")\n",
    "    user_job = loaded_graph.get_tensor_by_name(\"user_job:0\")\n",
    "    movie_id = loaded_graph.get_tensor_by_name(\"movie_id:0\")\n",
    "    movie_categories = loaded_graph.get_tensor_by_name(\"movie_categories:0\")\n",
    "    movie_titles = loaded_graph.get_tensor_by_name(\"movie_titles:0\")\n",
    "    targets = loaded_graph.get_tensor_by_name(\"targets:0\")\n",
    "    dropout_keep_prob = loaded_graph.get_tensor_by_name(\"dropout_keep_prob:0\")\n",
    "    lr = loaded_graph.get_tensor_by_name(\"LearningRate:0\")\n",
    "    #两种不同计算预测评分的方案使用不同的name获取tensor inference\n",
    "#     inference = loaded_graph.get_tensor_by_name(\"inference/inference/BiasAdd:0\")\n",
    "    inference = loaded_graph.get_tensor_by_name(\"inference/ExpandDims:0\") # 之前是MatMul:0 因为inference代码修改了 这里也要修改 感谢网友 @清歌 指出问题\n",
    "    movie_combine_layer_flat = loaded_graph.get_tensor_by_name(\"movie_fc/Reshape:0\")\n",
    "    user_combine_layer_flat = loaded_graph.get_tensor_by_name(\"user_fc/Reshape:0\")\n",
    "    return uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference, movie_combine_layer_flat, user_combine_layer_flat\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 指定用户和电影进行评分\n",
    "这部分就是对网络做正向传播，计算得到预测的评分"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "def rating_movie(user_id_val, movie_id_val):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "    \n",
    "        # Get Tensors from loaded model\n",
    "        uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, inference,_, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "    \n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = movies.values[movieid2idx[movie_id_val]][2]\n",
    "    \n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = movies.values[movieid2idx[movie_id_val]][1]\n",
    "    \n",
    "        feed = {\n",
    "              uid: np.reshape(users.values[user_id_val-1][0], [1, 1]),\n",
    "              user_gender: np.reshape(users.values[user_id_val-1][1], [1, 1]),\n",
    "              user_age: np.reshape(users.values[user_id_val-1][2], [1, 1]),\n",
    "              user_job: np.reshape(users.values[user_id_val-1][3], [1, 1]),\n",
    "              movie_id: np.reshape(movies.values[movieid2idx[movie_id_val]][0], [1, 1]),\n",
    "              movie_categories: categories,  #x.take(6,1)\n",
    "              movie_titles: titles,  #x.take(5,1)\n",
    "              dropout_keep_prob: 1}\n",
    "    \n",
    "        # Get Prediction\n",
    "        inference_val = sess.run([inference], feed)  \n",
    "    \n",
    "        return (inference_val)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[array([[3.700735]], dtype=float32)]"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rating_movie(234, 1401)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成Movie特征矩阵\n",
    "将训练好的电影特征组合成电影特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n"
     ]
    }
   ],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "movie_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, movie_combine_layer_flat, __ = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in movies.values:\n",
    "        categories = np.zeros([1, 18])\n",
    "        categories[0] = item.take(2)\n",
    "\n",
    "        titles = np.zeros([1, sentences_size])\n",
    "        titles[0] = item.take(1)\n",
    "\n",
    "        feed = {\n",
    "            movie_id: np.reshape(item.take(0), [1, 1]),\n",
    "            movie_categories: categories,  #x.take(6,1)\n",
    "            movie_titles: titles,  #x.take(5,1)\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        movie_combine_layer_flat_val = sess.run([movie_combine_layer_flat], feed)  \n",
    "        movie_matrics.append(movie_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(movie_matrics).reshape(-1, 200)), open('movie_matrics.p', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "movie_matrics = pickle.load(open('movie_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 生成User特征矩阵\n",
    "将训练好的用户特征组合成用户特征矩阵并保存到本地"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n"
     ]
    }
   ],
   "source": [
    "loaded_graph = tf.Graph()  #\n",
    "users_matrics = []\n",
    "with tf.Session(graph=loaded_graph) as sess:  #\n",
    "    # Load saved model\n",
    "    loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "    loader.restore(sess, load_dir)\n",
    "\n",
    "    # Get Tensors from loaded model\n",
    "    uid, user_gender, user_age, user_job, movie_id, movie_categories, movie_titles, targets, lr, dropout_keep_prob, _, __,user_combine_layer_flat = get_tensors(loaded_graph)  #loaded_graph\n",
    "\n",
    "    for item in users.values:\n",
    "\n",
    "        feed = {\n",
    "            uid: np.reshape(item.take(0), [1, 1]),\n",
    "            user_gender: np.reshape(item.take(1), [1, 1]),\n",
    "            user_age: np.reshape(item.take(2), [1, 1]),\n",
    "            user_job: np.reshape(item.take(3), [1, 1]),\n",
    "            dropout_keep_prob: 1}\n",
    "\n",
    "        user_combine_layer_flat_val = sess.run([user_combine_layer_flat], feed)  \n",
    "        users_matrics.append(user_combine_layer_flat_val)\n",
    "\n",
    "pickle.dump((np.array(users_matrics).reshape(-1, 200)), open('users_matrics.p', 'wb'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "users_matrics = pickle.load(open('users_matrics.p', mode='rb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 开始推荐电影\n",
    "使用生产的用户特征矩阵和电影特征矩阵做电影推荐"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐同类型的电影\n",
    "思路是计算当前看的电影特征向量与整个电影特征矩阵的余弦相似度，取相似度最大的top_k个，这里加了些随机选择在里面，保证每次的推荐稍稍有些不同。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_same_type_movie(movie_id_val, top_k = 20):\n",
    "    \n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "        \n",
    "        norm_movie_matrics = tf.sqrt(tf.reduce_sum(tf.square(movie_matrics), 1, keep_dims=True))\n",
    "        normalized_movie_matrics = movie_matrics / norm_movie_matrics\n",
    "\n",
    "        #推荐同类型的电影\n",
    "        probs_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(normalized_movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "以下是给您的推荐：\n",
      "1123\n",
      "[1139 'Everything Relative (1996)' 'Drama']\n",
      "1380\n",
      "[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "426\n",
      "[430 'Calendar Girl (1993)' 'Drama']\n",
      "3125\n",
      "[3194 'Way We Were, The (1973)' 'Drama']\n",
      "795\n",
      "[805 'Time to Kill, A (1996)' 'Drama']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{426, 795, 1123, 1380, 3125}"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_same_type_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 推荐您喜欢的电影\n",
    "思路是使用用户特征向量与电影特征矩阵计算所有电影的评分，取评分最高的top_k个，同样加了些随机选择部分。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "def recommend_your_favorite_movie(user_id_val, top_k = 10):\n",
    "\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        #推荐您喜欢的电影\n",
    "        probs_embeddings = (users_matrics[user_id_val-1]).reshape([1, 200])\n",
    "\n",
    "        probs_similarity = tf.matmul(probs_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     print(sim.shape)\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "        \n",
    "    #     sim_norm = probs_norm_similarity.eval()\n",
    "    #     print((-sim_norm[0]).argsort()[0:top_k])\n",
    "    \n",
    "        print(\"以下是给您的推荐：\")\n",
    "        p = np.squeeze(sim)\n",
    "        p[np.argsort(p)[:-top_k]] = 0\n",
    "        p = p / np.sum(p)\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = np.random.choice(3883, 1, p=p)[0]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "\n",
    "        return results\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "以下是给您的推荐：\n",
      "523\n",
      "[527 \"Schindler's List (1993)\" 'Drama|War']\n",
      "1132\n",
      "[1148 'Wrong Trousers, The (1993)' 'Animation|Comedy']\n",
      "2836\n",
      "[2905 'Sanjuro (1962)' 'Action|Adventure']\n",
      "3228\n",
      "[3297 'With Byrd at the South Pole (1930)' 'Documentary']\n",
      "2495\n",
      "[2564 'Empty Mirror, The (1999)' 'Drama']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{523, 1132, 2495, 2836, 3228}"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_your_favorite_movie(234, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 看过这个电影的人还看了（喜欢）哪些电影\n",
    "- 首先选出喜欢某个电影的top_k个人，得到这几个人的用户特征向量。\n",
    "- 然后计算这几个人对所有电影的评分\n",
    "- 选择每个人评分最高的电影作为推荐\n",
    "- 同样加入了随机选择"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "\n",
    "def recommend_other_favorite_movie(movie_id_val, top_k = 20):\n",
    "    loaded_graph = tf.Graph()  #\n",
    "    with tf.Session(graph=loaded_graph) as sess:  #\n",
    "        # Load saved model\n",
    "        loader = tf.train.import_meta_graph(load_dir + '.meta')\n",
    "        loader.restore(sess, load_dir)\n",
    "\n",
    "        probs_movie_embeddings = (movie_matrics[movieid2idx[movie_id_val]]).reshape([1, 200])\n",
    "        probs_user_favorite_similarity = tf.matmul(probs_movie_embeddings, tf.transpose(users_matrics))\n",
    "        favorite_user_id = np.argsort(probs_user_favorite_similarity.eval())[0][-top_k:]\n",
    "    #     print(normalized_users_matrics.eval().shape)\n",
    "    #     print(probs_user_favorite_similarity.eval()[0][favorite_user_id])\n",
    "    #     print(favorite_user_id.shape)\n",
    "    \n",
    "        print(\"您看的电影是：{}\".format(movies_orig[movieid2idx[movie_id_val]]))\n",
    "        \n",
    "        print(\"喜欢看这个电影的人是：{}\".format(users_orig[favorite_user_id-1]))\n",
    "        probs_users_embeddings = (users_matrics[favorite_user_id-1]).reshape([-1, 200])\n",
    "        probs_similarity = tf.matmul(probs_users_embeddings, tf.transpose(movie_matrics))\n",
    "        sim = (probs_similarity.eval())\n",
    "    #     results = (-sim[0]).argsort()[0:top_k]\n",
    "    #     print(results)\n",
    "    \n",
    "    #     print(sim.shape)\n",
    "    #     print(np.argmax(sim, 1))\n",
    "        p = np.argmax(sim, 1)\n",
    "        print(\"喜欢看这个电影的人还喜欢看：\")\n",
    "\n",
    "        results = set()\n",
    "        while len(results) != 5:\n",
    "            c = p[random.randrange(top_k)]\n",
    "            results.add(c)\n",
    "        for val in (results):\n",
    "            print(val)\n",
    "            print(movies_orig[val])\n",
    "        \n",
    "        return results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "INFO:tensorflow:Restoring parameters from ./save\n",
      "您看的电影是：[1401 'Ghosts of Mississippi (1996)' 'Drama']\n",
      "喜欢看这个电影的人是：[[100 'M' 35 17]\n",
      " [763 'M' 18 10]\n",
      " [3557 'M' 18 5]\n",
      " [4200 'M' 45 7]\n",
      " [3780 'M' 1 0]\n",
      " [5005 'M' 45 16]\n",
      " [3833 'M' 25 1]\n",
      " [4849 'F' 18 4]\n",
      " [5143 'M' 18 4]\n",
      " [4903 'M' 35 12]\n",
      " [3703 'M' 18 12]\n",
      " [5567 'M' 50 3]\n",
      " [3901 'M' 18 14]\n",
      " [4043 'F' 25 15]\n",
      " [3031 'M' 18 4]\n",
      " [1855 'M' 18 4]\n",
      " [4718 'M' 35 7]\n",
      " [1763 'M' 35 7]\n",
      " [2338 'M' 45 17]\n",
      " [1644 'M' 18 12]]\n",
      "喜欢看这个电影的人还喜欢看：\n",
      "847\n",
      "[858 'Godfather, The (1972)' 'Action|Crime|Drama']\n",
      "49\n",
      "[50 'Usual Suspects, The (1995)' 'Crime|Thriller']\n",
      "1178\n",
      "[1196 'Star Wars: Episode V - The Empire Strikes Back (1980)'\n",
      " 'Action|Adventure|Drama|Sci-Fi|War']\n",
      "315\n",
      "[318 'Shawshank Redemption, The (1994)' 'Drama']\n",
      "3228\n",
      "[3297 'With Byrd at the South Pole (1930)' 'Documentary']\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{49, 315, 847, 1178, 3228}"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "recommend_other_favorite_movie(1401, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 结论\n",
    "\n",
    "以上就是实现的常用的推荐功能，将网络模型作为回归问题进行训练，得到训练好的用户特征矩阵和电影特征矩阵进行推荐。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## 扩展阅读\n",
    "如果你对个性化推荐感兴趣，以下资料建议你看看："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [`Understanding Convolutional Neural Networks for NLP`](http://www.wildml.com/2015/11/understanding-convolutional-neural-networks-for-nlp/)\n",
    "\n",
    "- [`Convolutional Neural Networks for Sentence Classification`](https://github.com/yoonkim/CNN_sentence)\n",
    "\n",
    "- [`利用TensorFlow实现卷积神经网络做文本分类`](http://www.jianshu.com/p/ed3eac3dcb39?from=singlemessage)\n",
    "\n",
    "- [`Convolutional Neural Network for Text Classification in Tensorflow`](https://github.com/dennybritz/cnn-text-classification-tf)\n",
    "\n",
    "- [`SVD Implement Recommendation systems`](https://github.com/songgc/TF-recomm)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "今天的分享就到这里，请多指教！"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
